But other people have heard about CEP engines. They've been to web sites of CEP vendors. They've read articles and blogs about CEP. Yet they are still utterly confused as to what a CEP Engine is and what it's good for. To be honest, you can't blame these people either, and here is why.

I am yet to see a simple explanation of what a CEP engine is. Some folks in the CEP community (no finger pointing, please) claim that CEP is something so wonderfully complex that no CEP engine can really do it yet. I can see how this might be confusing! Some CEP vendors focus a lot more on specific applications, especially in capital markets. Some (ok, you can point at me) focus on enumerating specific CEP Design Patterns, which makes people believe that they have to understand all these design patterns to get the basic idea of what a CEP engine is.

Nothing can be further from the truth. Let me present a one sentence definition of what a CEP engine is:

A CEP engine is a platform that makes it easy to write and deploy applications that process and analyze real-time data.

That's it! This simple definition covers the Coral8 CEP engine, as well as all other CEP engines built by today's vendors. Now, let's analyze the definition word by word:

A CEP engine is a platform... This addresses a common misconception. A CEP engine is not an application! It does not magically do Risk monitoring, Intrusion detection, Fraud prevention, Web site ad targeting, or anything of this nature out of the box. All these, and many other applications can be written on top of a good CEP engine, but the engine by itself does not do this any more than a database knows how to query itself.

...that makes it easy... This is key. All the wonderful CEP applications people talk about can be written without CEP engines, using traditional languages such as C++ or Java. Anybody who tells you otherwise is not being honest. But a CEP engine makes it a whole lot easier to write these applications! The saving in time, resources, and therefore money, according to our many customers, is easily a factor of 10!

...to write and deploy... A CEP engine is not just a development tool, but also a scalable infrastructure for deploying CEP applications. Never forget about this. If you are going to analyze 100,000s of events (pieces of real-time data) per second with sub-millisecond latency, you need a special piece of infrastructure - a CEP engine.

...applications that process and analyze real-time data. This is the key. A CEP engine is all about real-time data. If you want to collect, filter, aggregate, enrich, transform, correlate, and store real-time data, or do pretty much anything else with it, then you may benefit from a CEP engine. In fact, I would be really curious to find out about applications that need to deal with real-time data that cannot utilize a CEP engine.

I hope this makes sense. Of course, in this post I have not explained how a CEP engine works with real-time data, how it is programmed, what all its wonderful features may be, and so on. My goal was to provide a one sentence definition of what a CEP engine is, period. Whether I failed or succeeded, I would like to hear from you.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/webinar-third-generation-algorithmic-trading-execution-complex-event-processing 2008-10-07T10:32:10-07:00 2008-10-07T10:32:10-07:00 Mark

(Yes, I'm trying to learn from the best, in this case from the very excellent blog by Opher Etzion which has very cool pictures in every post.)

A lot has been written on the various CEP accomplishments over the past five years, but also about the challenges that lie ahead. Personally, when I think about the current state of CEP, I can't help but think about the famous words of Winston Churchill: "This is not the end. This is not even the beginning of the end. But, it is, perhaps the end of the beginning."

It's hard to disagree with this statement. If you compare CEP to a truly mature market, e.g., the market for relational databases, then it's easy to see that CEP has a long way to go. Yet if you compare the state of the CEP to, say the year 2003, when Coral8 was founded, it's hard to argue that the CEP community has not made a lot of progress.

Then why is this debate raging? One may argue that the people who claim that CEP is mature are mainly CEP vendors, and it's in their interest to do so. But I think the answer is a bit different.

As an amateur economist (isn't everyone these days?), I like to think about what makes markets efficient or inefficient. One concept that comes up all the time is information asymmetry. Information asymmetry refers to situations when buyers and sellers in a market have different levels of access to information. Consider, for example, the markets for used cars or health insurance. In the first case, the seller of the car knows a lot more about the car than the buyer. In the second case, the buyer of health insurance knows a lot more about his or her health than the seller. This makes these markets highly inefficient, as anybody who's tried to buy a used car or health insurance will readily testify.

What does it have to do with CEP? A lot. I claim that CEP vendors know a lot more about CEP success than the general public does. I'm sure you've seen a table compiled by Tim Bass listing the number of publicly announced customers for each CEP vendor. The table would be pretty funny, if it was not so sad. According to this table, no CEP vendor has more than 4 customers! This is hardly a sign of a mature market.

I don't want to speak for every CEP vendor out there, and I don't want to name names or exact numbers, but I know for a fact that every major CEP vendor has several dozen paying customers. Not thousands of customers, not hundreds of customers, but several dozen - for sure. I believe this is why vendors have a more positive view on CEP's maturity - they see the use cases and success stories that support the view!

As I mentioned earlier, information asymmetry is not a good thing. It prevents the CEP market from developing faster, as prospects have a skewed view of how mature the products really are. So why do existing customer resist telling their stories, and even resist being named? "We believe that the use of Corla8 gives us a strategic advantage over our competitors. Why would we want to clue them in?" - this is what we, and probably other vendors, hear over and over and over.

It's a bit hard to argue with this logic, but again this represents a problem for accelerating the growth of CEP. I think identifying the problem is half the challenge. But what can we, as a community, do to solve this problem? Perhaps we can learn something from other communities? If anybody has any great stories or suggestions, I'd like to hear them.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/what-makes-programming-language-successful 2008-05-29T12:11:51-07:00 2008-05-29T12:11:51-07:00 Mark

What Makes a Programming Language Successful?" is the subject of a post on Slashdot today. It links to an article that is probably quite illuminating, but I have not read it: the web site is heavily slashdotted (overwhelmed).

But there are a number of comments posted on this page, and they have typical Slashdot tags: "Interesting", "Insightful", "Informative". One of the comments is marked "Funny", but I think it is anything but. The comment answers the question "What Makes a Programming Language Successful?" with a simple one liner: "Those who don't know how to use it."

I think this deserves at least "Insightful," and this is why. You remember that a few posts ago I wrote about what makes a Coral8 expert, and we had a lively discussion on this topic, both in this blog and others. I still stand by everything I said there, but that was the post about experts. Most users of every programming language, including CCL, are not, and will never be experts.

There is no shame in not being an expert. Every person usually learns just enough of various skills to accomplish whatever tasks he or she needs. And this is what a programming language, or any other tools, should be good at - helping people, especially beginners, accomplish whatever tasks they need.

So what's the most important quality that the programming language should have? As I said, I have not read the article, so I'll give you my own answer instead. I strognly believe that first and foremost, the programming language and the programming model should look familiar.

This is one reason why CCL is based on SQL. Is SQL the only language that you could base a CEP language on? No. It's certainly a good choice, I'd even argue it's the best choice, but it's not the only one.

But whatever you choose, you must make sure that the language looks and feels familiar. Beginner users do not have time to study lengthy manuals to learn a new language, even if this is the most elegant language in the world. The language either "makes sense" to them, or it does not. In this sense, the people who don't know the programming language, or who don't know it very well, are truly the ones who make it successful. Remember, even every expert starts as a beginner.

At Coral8, we try to keep this in mind as we are building our products. We like it that Coral8 and CCL get great reviews from CEP experts, but we also like it that somebody who has never heard of CEP can download our product and build their first real CEP application within a few days, without much, if any, help. We hear about this happening every day. Their CCL may not be the most elegant or optimal, but it works. As long as it keeps happening, I know we are doing something right here.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/complexity-scorecard 2008-04-16T15:55:40-07:00 2008-04-16T16:15:42-07:00 Mark

More on CEP and Complexity I made a claim that for the simplest CEP applications, one does not really need Coral8, or any CEP engine for that matter. You can write your application in C++ or Java just fine. For medium complexity applications, most commercial or open source engines would do. But for sophisticated applications, you need to go with a truly high-end CEP engine. But, a prospect asked me yesterday, how do I know where my application lands on the sophistication scale? As a public service, here I provide a scorecard to judge how sophisticated your CEP application is.

For each of the ten questions, please choose one answer. The more points you score, the more sophisticated your application is.

What is the combined data rate you need to support?

• Less than 1 event/sec: 0 (Just curious, why are you reading this blog?)
• 1-100 events/sec: 1
• 100-1000 events/sec: 3
• 1000-10,000 events/sec: 4
• 10,000+ events/sec: 5

What is the event processing latency you need to guarantee?

• Minutes/Hours/Days: 0 (Still not sure why you are reading this)
• Seconds: 1
• 100-1000 milliseconds: 2
• 10-100 milliseconds: 3
• 1-10 milliseconds: 4
• Less than 1 millisecond: 5

How many data streams do you have?

• One stream: 1
• More than one, but I don't need to synchronize events across streams: 2
• More than one, and I need to synchronize events across streams, and handle delayed and out-of-order events: 5

How large/complex are your input events?

• Small flat events (1-10 fields): 1
• Large flat events (10+ fields): 2
• Large non-flat / hierarchical / XML events: 5

What do most of your queries look like?

• Filtering and single-event transformations: 1
• Aggregation over different kinds of windows: 3
• Joins and state management: 4
• Event Pattern Matching: 5

How many queries do you have?

• 1-10: 1
• 10-100: 3
• 100+: 5

Do you need to interface with databases?

• No: 0
• Infrequent reads or writes: 2
• Frequent reads/writes, but no read/write caching: 3
• Frequent reads/writes, need basic caching: 4
• Frequent reads/writes, need granular on-demand row caching: 5

Do you need to scale beyond one CPU?

• No: 0
• Yes, to multiple CPUs/cores: 2
• Yes, to several machines: 4
• Yes, to a large cluster: 5

What are your High-Availability / Data Persistence requirements?

• None: 0 (I don't believe you, but that's ok)
• I need to recover from failures, but I don't care if I lose data: 2
• I don't want to lose data, but I'm ok with losing a few events here and there: 4
• I never, ever, want to lose a single event!: 5

What kind of interface do you need for business users?

• None: 0 (Who is paying for your project?)
• They are ok with static reports: 1
• They want real-time dashboards: 3
• They want real-time dashboards, and they want to configure dashboards and actions dynamically: 5

Now, add up your points:

• Less than 15: Your application is not too sophisticated. You don't really need a CEP engine right now. A CEP Engine will still save you time and money, especially over the long term, but you don't really need one
• 15-30: Your application is of medium sophistication. Using a CEP engine is highly advised, but the good news is that most CEP engines should be able to handle it.
• More than 30: You application is sophisticated. Please choose your CEP engine very carefully, as most CEP engines out there will not be able to handle it.

That's it! Of course, this scorecard is very approximate, and I have not covered a lot of areas: adapters to external systems, SDK, deployment, management, security, determinism, etc. But I hope this is still useful. Please let me know if you have any feedback or questions.

Mark Tsimelzon, President & CTO, Coral8 ]]>

More on CEP and Complexity I made a claim that for the simplest CEP applications, one does not really need Coral8, or any CEP engine for that matter. You can write your application in C++ or Java just fine. For medium complexity applications, most commercial or open source engines would do. But for sophisticated applications, you need to go with a truly high-end CEP engine. But, a prospect asked me yesterday, how do I know where my application lands on the sophistication scale? As a public service, here I provide a scorecard to judge how sophisticated your CEP application is.

For each of the ten questions, please choose one answer. The more points you score, the more sophisticated your application is.

What is the combined data rate you need to support?

• Less than 1 event/sec: 0 (Just curious, why are you reading this blog?)
• 1-100 events/sec: 1
• 100-1000 events/sec: 3
• 1000-10,000 events/sec: 4
• 10,000+ events/sec: 5

What is the event processing latency you need to guarantee?

• Minutes/Hours/Days: 0 (Still not sure why you are reading this)
• Seconds: 1
• 100-1000 milliseconds: 2
• 10-100 milliseconds: 3
• 1-10 milliseconds: 4
• Less than 1 millisecond: 5

How many data streams do you have?

• One stream: 1
• More than one, but I don't need to synchronize events across streams: 2
• More than one, and I need to synchronize events across streams, and handle delayed and out-of-order events: 5

How large/complex are your input events?

• Small flat events (1-10 fields): 1
• Large flat events (10+ fields): 2
• Large non-flat / hierarchical / XML events: 5

What do most of your queries look like?

• Filtering and single-event transformations: 1
• Aggregation over different kinds of windows: 3
• Joins and state management: 4
• Event Pattern Matching: 5

How many queries do you have?

• 1-10: 1
• 10-100: 3
• 100+: 5

Do you need to interface with databases?

• No: 0
• Infrequent reads or writes: 2
• Frequent reads/writes, but no read/write caching: 3
• Frequent reads/writes, need basic caching: 4
• Frequent reads/writes, need granular on-demand row caching: 5

Do you need to scale beyond one CPU?

• No: 0
• Yes, to multiple CPUs/cores: 2
• Yes, to several machines: 4
• Yes, to a large cluster: 5

What are your High-Availability / Data Persistence requirements?

• None: 0 (I don't believe you, but that's ok)
• I need to recover from failures, but I don't care if I lose data: 2
• I don't want to lose data, but I'm ok with losing a few events here and there: 4
• I never, ever, want to lose a single event!: 5

What kind of interface do you need for business users?

• None: 0 (Who is paying for your project?)
• They are ok with static reports: 1
• They want real-time dashboards: 3
• They want real-time dashboards, and they want to configure dashboards and actions dynamically: 5

Now, add up your points:

• Less than 15: Your application is not too sophisticated. You don't really need a CEP engine right now. A CEP Engine will still save you time and money, especially over the long term, but you don't really need one
• 15-30: Your application is of medium sophistication. Using a CEP engine is highly advised, but the good news is that most CEP engines should be able to handle it.
• More than 30: You application is sophisticated. Please choose your CEP engine very carefully, as most CEP engines out there will not be able to handle it.

That's it! Of course, this scorecard is very approximate, and I have not covered a lot of areas: adapters to external systems, SDK, deployment, management, security, determinism, etc. But I hope this is still useful. Please let me know if you have any feedback or questions.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/more-cep-and-complexity 2008-04-05T19:17:00-07:00 2008-04-16T16:11:34-07:00 Mark

last post I was hoping it'd start a discussion going. So I'm very grateful to Greg Reemler for his reply. While Greg makes some interesting points, I, naturally, respectfully disagree with some of them.

It's easy to argue against small points in Greg's post. For example, Greg writes "Coral8 sounds more like a lab tool for the engineering department of Caltech or Stanford than a tool for everyday business users." Well, everybody knows that Coral8 Engine, as well as other CEP platforms, as opposed to CEP applications, are tools for developers, and not for business users.

Or, it's easy to point to other very successful examples of enterprise software products that are more complex than the Coral8 Engine. But I don't want to do this. Because Greg has an excellent point: if a product is needlessly complex, nobody is going to use it!

The place where Greg and I disagree is whether our product is needlessly complex. Here is my claim: the list in the previous post covers the features and skills needed to write sophisticated real-world CEP applications. Those are the applications that process tens and hundreds of thousands of events per second, coming from a number of very different data sources, integrating with databases, message buses, and numerous other third party systems, and performing highly complex analysis with millisecond latency. All this while being highly scalable, available, and secure.

Unfortunately, most of discussion around CEP today focuses on very simple use cases. Everybody publishes their implementation of VWAP (Volume-Weighted Average Price) and some benchmarks for VWAP. Well, if CEP was limited to such use cases, people would be right to consider CEP just hype. Writing a simple CEP application is indeed quite easy in C or Java, as Greg correctly notes. It's when the complexity of the application grows that CEP engines start to shine.

How do I know this? We have about 50+ customers now, and good 90% of them had started writing their applications using traditional languages, such as C or Java. And they all were happy for a while. But then they needed to add more streams, more data sources, and more complex queries. They needed to talk to databases and to Web Services. They needed to handle XML events and complex event patterns. And most of all, they needed to make sure their application would never fail.

That's when they start looking around, find this whole field of CEP, and realize that the Coral8 Engine is indeed a great choice for writing highly sophisticated CEP applications. Our product makes accomplishing all the tasks listed in the last post easy. Is there a learning curve? Sure. It does not take a week, as Greg suggests, to learn each of the 60 topics. It takes maybe an hour, give or take. Which means that in under two weeks customers can become true CEP experts, if their applications demand this level of expertise. If you consider what they get for their effort, you'll understand why they feel the effort is well worth it.

So, if I may offer Greg some friendly advice, here it is: if your CEP application is simple, by all means use your C or Java programmers. I'm sure they'll do a fine job. But if or when you realize that the requirements are more complex, feel free to give us a call. We'll be happy to talk to you and your team about our product.

Mark Tsimelzon, President & CTO, Coral8 ]]>

last post I was hoping it'd start a discussion going. So I'm very grateful to Greg Reemler for his reply. While Greg makes some interesting points, I, naturally, respectfully disagree with some of them.

It's easy to argue against small points in Greg's post. For example, Greg writes "Coral8 sounds more like a lab tool for the engineering department of Caltech or Stanford than a tool for everyday business users." Well, everybody knows that Coral8 Engine, as well as other CEP platforms, as opposed to CEP applications, are tools for developers, and not for business users.

Or, it's easy to point to other very successful examples of enterprise software products that are more complex than the Coral8 Engine. But I don't want to do this. Because Greg has an excellent point: if a product is needlessly complex, nobody is going to use it!

The place where Greg and I disagree is whether our product is needlessly complex. Here is my claim: the list in the previous post covers the features and skills needed to write sophisticated real-world CEP applications. Those are the applications that process tens and hundreds of thousands of events per second, coming from a number of very different data sources, integrating with databases, message buses, and numerous other third party systems, and performing highly complex analysis with millisecond latency. All this while being highly scalable, available, and secure.

Unfortunately, most of discussion around CEP today focuses on very simple use cases. Everybody publishes their implementation of VWAP (Volume-Weighted Average Price) and some benchmarks for VWAP. Well, if CEP was limited to such use cases, people would be right to consider CEP just hype. Writing a simple CEP application is indeed quite easy in C or Java, as Greg correctly notes. It's when the complexity of the application grows that CEP engines start to shine.

How do I know this? We have about 50+ customers now, and good 90% of them had started writing their applications using traditional languages, such as C or Java. And they all were happy for a while. But then they needed to add more streams, more data sources, and more complex queries. They needed to talk to databases and to Web Services. They needed to handle XML events and complex event patterns. And most of all, they needed to make sure their application would never fail.

That's when they start looking around, find this whole field of CEP, and realize that the Coral8 Engine is indeed a great choice for writing highly sophisticated CEP applications. Our product makes accomplishing all the tasks listed in the last post easy. Is there a learning curve? Sure. It does not take a week, as Greg suggests, to learn each of the 60 topics. It takes maybe an hour, give or take. Which means that in under two weeks customers can become true CEP experts, if their applications demand this level of expertise. If you consider what they get for their effort, you'll understand why they feel the effort is well worth it.

So, if I may offer Greg some friendly advice, here it is: if your CEP application is simple, by all means use your C or Java programmers. I'm sure they'll do a fine job. But if or when you realize that the requirements are more complex, feel free to give us a call. We'll be happy to talk to you and your team about our product.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/what-makes-coral8-expert 2008-04-03T14:41:59-07:00 2008-04-03T22:02:09-07:00 Mark

At the same time, many production Coral8 applications are quite complex, and to build these powerful applications one needs a certain amount of knowledge and expertise. This is true of any CEP engine or any powerful enterprise system. So I am asked more and more often by customers and partners: What makes a Coral8 expert? How do we train our employees to best take advantage of Coral8?

We don't have a formal "Coral8 Certification" course yet, but I thought I'd put together a list of areas where one needs to develop expertise to be considered a true Coral8 expert. Of course, the list is not meant to scare anybody away. Plenty of people use Coral8 successfully without understanding 20% of these topics. But if you want to become a Coral8 Systems Integrator Partner, then here is what you need to know:

CCL

Windows

• Different kinds of windows (time-based, row-based, sliding, jumping, largest/smallest, partitioned (PER), multi-policy, bucketc, etc.)
• Named windows and implicit windows
• Using aggregations and windows
• Using windows to keep and manage state
• Querying windows (public windows)

Joins

• Stream to Window join
• Window to Window joins
• Outer joins

Patterns

• Basic pattern operators ([], ",", ||, &&, !)
• Nested patterns
• Causality tracking (XMLPatternMatch())

XML

• XML datatype
• Working with XML: parsing, constructing, transforming, joining, etc.

Modules

• Using modules for abstraction and code reuse
• Module interface: input/output streams and parameters

Procedural features

• functions
• variables
• conditionals
• loops

User-Defined Functions

• Calling scalar user-defined functions written C
• Calling aggregation user-defined functions written in C
• Calling RPC style user-defined functions

Design patterns and basic recipes

• Design Patterns http://www.coral8.com/system/files/assets/pdf/Coral8DesignPatterns.pdf
• Cookbook http://www.coral8.com/system/files/assets/pdf/5.2.0/CCL%20Cookbook.pdf

Integrations with databases

• Reading data from a database using database subqueries in CCL
• Writing data into a database using EXECUTE STATEMENT
• Configuring database connections
• Caching
• Performance and reliability options
• Using adapters to play back data from databases

Messaging layer

• stream URL
• publish/subscribe
• timestamp assignment: message timestamp vs server timestamp
• delayed messages and synchronization across multiple streams
• out-of-order message handling

Adapters

• Adapters shipped with the product (message bus, database, file, socket, specialty, etc.)
• In-process vs. out-of-process adapters
• Writing adapter in C/C++, Java, .NET, Perl, Python

Enterprise features

• State Persistence
• High Availability: different options
• Guaranteed Message Delivery
• Combining all Persistence, HA, and GD to meet the requirements
• Clustering for performance and failover
• Security: SSL encryption, authentication, authorization, integration with LDAP and other systems

Performance

• How to maximize throughput of Coral8 Engine
• How to minimize latency of Coral8 Engine
• How to write high-performance adapters
• How to take advantage of multiple CPUs or cores
• Hot to take advantage of clustering by splitting streams or queries across machines.

Studio

• Creating CCL projects and modules
• Using environment files to manage connections to workspaces
• Monitoring server performance, throughput, latency and resources through the Studio

Development processes

• Using multiple workspaces for development, QA, production, etc.
• Using source control
• Upgrading the server & studio from one version to the next

Command-line tools and their options

• c8_compiler to compile CCL
• c8_client to manage the server, pub/sub to streams, submit queries dynamically, etc.

Portal

• Setting up the Portal
• Creating query templates for use with the Portal
• Different visualization options
• Actions

That's it! Not too scary, is it?

Mark Tsimelzon, President & CTO, Coral8 ]]>

At the same time, many production Coral8 applications are quite complex, and to build these powerful applications one needs a certain amount of knowledge and expertise. This is true of any CEP engine or any powerful enterprise system. So I am asked more and more often by customers and partners: What makes a Coral8 expert? How do we train our employees to best take advantage of Coral8?

We don't have a formal "Coral8 Certification" course yet, but I thought I'd put together a list of areas where one needs to develop expertise to be considered a true Coral8 expert. Of course, the list is not meant to scare anybody away. Plenty of people use Coral8 successfully without understanding 20% of these topics. But if you want to become a Coral8 Systems Integrator Partner, then here is what you need to know:

CCL

Windows

• Different kinds of windows (time-based, row-based, sliding, jumping, largest/smallest, partitioned (PER), multi-policy, bucketc, etc.)
• Named windows and implicit windows
• Using aggregations and windows
• Using windows to keep and manage state
• Querying windows (public windows)

Joins

• Stream to Window join
• Window to Window joins
• Outer joins

Patterns

• Basic pattern operators ([], ",", ||, &&, !)
• Nested patterns
• Causality tracking (XMLPatternMatch())

XML

• XML datatype
• Working with XML: parsing, constructing, transforming, joining, etc.

Modules

• Using modules for abstraction and code reuse
• Module interface: input/output streams and parameters

Procedural features

• functions
• variables
• conditionals
• loops

User-Defined Functions

• Calling scalar user-defined functions written C
• Calling aggregation user-defined functions written in C
• Calling RPC style user-defined functions

Design patterns and basic recipes

• Design Patterns http://www.coral8.com/system/files/assets/pdf/Coral8DesignPatterns.pdf
• Cookbook http://www.coral8.com/system/files/assets/pdf/5.2.0/CCL%20Cookbook.pdf

Integrations with databases

• Reading data from a database using database subqueries in CCL
• Writing data into a database using EXECUTE STATEMENT
• Configuring database connections
• Caching
• Performance and reliability options
• Using adapters to play back data from databases

Messaging layer

• stream URL
• publish/subscribe
• timestamp assignment: message timestamp vs server timestamp
• delayed messages and synchronization across multiple streams
• out-of-order message handling

Adapters

• Adapters shipped with the product (message bus, database, file, socket, specialty, etc.)
• In-process vs. out-of-process adapters
• Writing adapter in C/C++, Java, .NET, Perl, Python

Enterprise features

• State Persistence
• High Availability: different options
• Guaranteed Message Delivery
• Combining all Persistence, HA, and GD to meet the requirements
• Clustering for performance and failover
• Security: SSL encryption, authentication, authorization, integration with LDAP and other systems

Performance

• How to maximize throughput of Coral8 Engine
• How to minimize latency of Coral8 Engine
• How to write high-performance adapters
• How to take advantage of multiple CPUs or cores
• Hot to take advantage of clustering by splitting streams or queries across machines.

Studio

• Creating CCL projects and modules
• Using environment files to manage connections to workspaces
• Monitoring server performance, throughput, latency and resources through the Studio

Development processes

• Using multiple workspaces for development, QA, production, etc.
• Using source control
• Upgrading the server & studio from one version to the next

Command-line tools and their options

• c8_compiler to compile CCL
• c8_client to manage the server, pub/sub to streams, submit queries dynamically, etc.

Portal

• Setting up the Portal
• Creating query templates for use with the Portal
• Different visualization options
• Actions

That's it! Not too scary, is it?

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/deployed-globally 2008-03-10T11:06:30-07:00 2008-03-10T11:06:30-07:00 Mark

Of course, we've had customer in the U.S., Europe, and Asia for a while. Some of them are financial services institutions, some are large web sites, some build or operate large sensor/RFID networks, and so on. But we also have a couple of OEM customers in Australia: they are in the network/server/app management space. And we have a customer in South America, who manages a large power grid using Coral8. And now with this bank in South Africa, we have Africa covered!

Just to be clear, I am talking about customers that have purchased the Coral8 product, rather than developers who've downloaded the software from our web site. If we count developers, the count will be in thousands. I don't have the exact count, and some don't even tell us where they come from, but I'm sure they come from at least 50 countries, from all over the globe.

Although I must admit, we have neither customers nor developers coming from Antarctica. Does anybody know what kind of CEP applications penguins are interested in?

Mark Tsimelzon, President & CTO, Coral8 ]]>

Of course, we've had customer in the U.S., Europe, and Asia for a while. Some of them are financial services institutions, some are large web sites, some build or operate large sensor/RFID networks, and so on. But we also have a couple of OEM customers in Australia: they are in the network/server/app management space. And we have a customer in South America, who manages a large power grid using Coral8. And now with this bank in South Africa, we have Africa covered!

Just to be clear, I am talking about customers that have purchased the Coral8 product, rather than developers who've downloaded the software from our web site. If we count developers, the count will be in thousands. I don't have the exact count, and some don't even tell us where they come from, but I'm sure they come from at least 50 countries, from all over the globe.

Although I must admit, we have neither customers nor developers coming from Antarctica. Does anybody know what kind of CEP applications penguins are interested in?

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/premature-optimization 2008-02-11T19:43:40-08:00 2008-02-11T19:43:40-08:00 Mark

Now, this is a great coffee shop, and the workers usually know what they are doing. Normally, there are two people filling orders, and in this case indeed starting the drinks first would be faster. But in a different situation with only one worker present, the guy's intuition was wrong. The algorithm that was great for two "processors", was not so great for one.

What's the moral of the story? We all like optimizations, but faced with new unfamiliar circumstances, our intuition often fails us. And what does it have to do with CEP? CEP is still fairly new to most people who build CEP applications. I wrote in my previous post that one of the first questions we hear from people is "how fast is your engine". Now, you will not be surprised that one of the more common follow up questions is "Will I make it faster if I do X"? Often, this is before the requirements are even firmed up, let alone the application is coded and profiled!

Now, CEP engines are complex beasts, and they contain numerous internal optimizations. The Coral8 Engine, for example, contains a large number of optimizations, some of which, e.g. push filter to before joins are familiar to SQL developers. Other optimizations, however, are completely new: automatic data indexing, optimized memory management to conserve space, sophisticated data caching, a unique threading model to limit context switches and take advantage of multiple CPU cores, a very lightweight messaging layer, and many others. We are taking great care to optimize throughput, latency, and resource consumption.

The flip side of this, however, is that it may not be immediately obvious to someone how fast their first Coral8 application will run on our engine. Therefore, our advice to all Coral8 developers is very simple: don't worry about performance at first. Write your application, or a representative portion of it first, and then run it. The Coral8 Engine will tell you how much CPU or memory it takes, what the throughput and latency are, and will help you pinpoint the problems. Then, if necessary, you can start your optimizations. But chances are, you won't have to: The vast majority of our customers are surprised at how fast their application is once they write it.

Anyway, I know I am not saying anything new here. Don Knuth said over three decades ago that premature optimization is the root of all evil. This is especially true when you are working in a new and unfamiliar environment. The SQL-based language used by Coral8 and other CEP engines looks familiar, and indeed the similarity with SQL makes programming simpler. But it helps to remember that the implementation of any CEP engine is very different from that of relational database, and is already highly optimized for CEP applications to begin with. So it's best to build the application first, measure the performance, and then start worrying about optimizing it if necessary.

Mark Tsimelzon
President & CTO, Coral8 ]]>

Now, this is a great coffee shop, and the workers usually know what they are doing. Normally, there are two people filling orders, and in this case indeed starting the drinks first would be faster. But in a different situation with only one worker present, the guy's intuition was wrong. The algorithm that was great for two "processors", was not so great for one.

What's the moral of the story? We all like optimizations, but faced with new unfamiliar circumstances, our intuition often fails us. And what does it have to do with CEP? CEP is still fairly new to most people who build CEP applications. I wrote in my previous post that one of the first questions we hear from people is "how fast is your engine". Now, you will not be surprised that one of the more common follow up questions is "Will I make it faster if I do X"? Often, this is before the requirements are even firmed up, let alone the application is coded and profiled!

Now, CEP engines are complex beasts, and they contain numerous internal optimizations. The Coral8 Engine, for example, contains a large number of optimizations, some of which, e.g. push filter to before joins are familiar to SQL developers. Other optimizations, however, are completely new: automatic data indexing, optimized memory management to conserve space, sophisticated data caching, a unique threading model to limit context switches and take advantage of multiple CPU cores, a very lightweight messaging layer, and many others. We are taking great care to optimize throughput, latency, and resource consumption.

The flip side of this, however, is that it may not be immediately obvious to someone how fast their first Coral8 application will run on our engine. Therefore, our advice to all Coral8 developers is very simple: don't worry about performance at first. Write your application, or a representative portion of it first, and then run it. The Coral8 Engine will tell you how much CPU or memory it takes, what the throughput and latency are, and will help you pinpoint the problems. Then, if necessary, you can start your optimizations. But chances are, you won't have to: The vast majority of our customers are surprised at how fast their application is once they write it.

Anyway, I know I am not saying anything new here. Don Knuth said over three decades ago that premature optimization is the root of all evil. This is especially true when you are working in a new and unfamiliar environment. The SQL-based language used by Coral8 and other CEP engines looks familiar, and indeed the similarity with SQL makes programming simpler. But it helps to remember that the implementation of any CEP engine is very different from that of relational database, and is already highly optimized for CEP applications to begin with. So it's best to build the application first, measure the performance, and then start worrying about optimizing it if necessary.

Mark Tsimelzon
President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/something-funny-start-year 2008-01-11T14:38:19-08:00 2008-01-11T15:02:01-08:00 Mark

(I have no idea whether it's legal to post it here, but hopefully it's ok)

I am not going to explain here why most performance numbers and benchmarks should be treated with extreme suspicion. They typically do not fully specify what's being computed, what the incoming events look like, how they get into the engine, what processing options (guaranteed delivery, failover, etc.) are enabled, what exactly is being measured, and so on. This has already been explained in other places.

I'll just note that we get a fair number of customer inquires that look like this "Hello, could you, please, tell us 1) how much your engine costs? 2) how many messages it can process per second?" That's it! Without any explanation of what they want to do, what the application looks like, even what domain it is in. Nothing!

I wonder if they send such requests to multiple CEP vendors, and then cleverly divide the second number by the first, to establish a CEP "message/$" rating for each vendor?

Mark Tsimelzon
President & CTO, Coral8 ]]>

(I have no idea whether it's legal to post it here, but hopefully it's ok)

I am not going to explain here why most performance numbers and benchmarks should be treated with extreme suspicion. They typically do not fully specify what's being computed, what the incoming events look like, how they get into the engine, what processing options (guaranteed delivery, failover, etc.) are enabled, what exactly is being measured, and so on. This has already been explained in other places.

I'll just note that we get a fair number of customer inquires that look like this "Hello, could you, please, tell us 1) how much your engine costs? 2) how many messages it can process per second?" That's it! Without any explanation of what they want to do, what the application looks like, even what domain it is in. Nothing!

I wonder if they send such requests to multiple CEP vendors, and then cleverly divide the second number by the first, to establish a CEP "message/$" rating for each vendor?

Mark Tsimelzon
President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/cep-library-or-server-infrastructure 2007-12-20T20:16:10-08:00 2008-03-01T09:33:53-08:00 Mark

Yes, it is possible to link the Coral8 server into another applications. But I want to make a bold claim here. The future of CEP is not about being a cool event processing library embedded into individual applications. The future of CEP is a shared, scalable, and reliable enterprise-wide server infrastructure.

This should not be a bold claim. Think back to my favorite analogy, databases. It's true that there exists a market for embedded database libraries. For example, SQLite is a nice simple library that can be easily linked into C applications. There are many others. But this market is dwarfed by the market for database servers. Oracle, DB/2, SQL Server, Sybase - all the common databases are typically not embedded into an application, but rather run on a separate server or cluster of servers.

Why is this the case? Let's come back to the CEP world, where the advantages of an infrastructure over a library are even more obvious.

Sharing

Let's start with sharing. Even though people typically start with one application, they quickly start deploying more and more. We have customers that have deployed dozens of applications by now! These applications all talk to each other via event streams. Moreover, these applications may dynamically deploy queries to each other. Rapidly integrating new applications is very hard if each application has to link in its own standalone CEP engine.

Scalability

What if the demands on a CEP applications grow, and a single server does not have enough CPU power to satisfy them? If the CEP engine is linked in as a library, there is not much one can do in this situation! On the other hand, a good CEP infrastructure can distribute event processing to multiple servers, balance the load, and satisfy the most stringent scalability demands.

Reliability

What happens if a CEP engine is linked in as a library, and the machine on which it runs dies a sudden death? Well, you've got a problem. You don't have another server standing by ready to take over. You can, of course, replicate both the application and the CEP engine on two machines, but you'll have to figure out how to synchronize the two copies, how to share state, etc. All these problems are already solved by a good server infrastructure product such as Coral8.

But wait...

So why do people even consider linking a CEP engine as a library? There are a few reasons, but they are somewhat misguided:

First, some people don't like the idea of starting a new process for a CEP server. While I sympathize with this sentiment, the price of starting a new process is usually not that high. Complex applications these days have many parts, and always never reside in one process anyway.

The second reason sounds more serious: latency! People think that if their application has to send events to a separate server, and then receive results back, the total processing latency will be a lot higher. Thus, they want to link the CEP server into their application.

This sort of makes sense, except for where do the events come from? In most cases, they don't come from the app, but from some third place: from a financial exchange, a web site, a sensor network, a message bus, etc. The right solution is not to send these events to the application, but directly to the CEP engine! Good CEP engines come with highly optimized adapters, some of them running in the process of the CEP engine itself. These adapters can process incoming events much faster than whatever an application can do! We have seen how some Coral8 adapters can process events in well under 100 microseconds! Why would an application want to re-implement these adapters? It makes little sense.

Finally, the third reason may be more emotional than anything else: Control. Some developers feel that if the engine is "just a library", they have more control over such things as threading model, memory management, and so on. Well, again, why would somebody want to worry about these issues, when CEP vendors spend years and years optimizing the performance of the CEP engines?

The CEP Cloud

I hope I've convinced you that the future of CEP belong to server-based infrastructure rather than CEP libraries. In fact, we have customers today who don't even care about our performance on a single server. They have a vision of running a cloud of CEP servers, and deploying hundreds and thousands of applications on this cloud. Of course, the cloud of these servers has to be completely self-managing. One does not need to worry about which server or servers are running a given app at any given moment, what happens when demand changes, what happens when servers go down, etc. We like this vision a lot. This is our vision, too. You just can't realize this vision via a simple library.

Mark Tsimelzon
President & CTO, Coral8 ]]>

Yes, it is possible to link the Coral8 server into another applications. But I want to make a bold claim here. The future of CEP is not about being a cool event processing library embedded into individual applications. The future of CEP is a shared, scalable, and reliable enterprise-wide server infrastructure.

This should not be a bold claim. Think back to my favorite analogy, databases. It's true that there exists a market for embedded database libraries. For example, SQLite is a nice simple library that can be easily linked into C applications. There are many others. But this market is dwarfed by the market for database servers. Oracle, DB/2, SQL Server, Sybase - all the common databases are typically not embedded into an application, but rather run on a separate server or cluster of servers.

Why is this the case? Let's come back to the CEP world, where the advantages of an infrastructure over a library are even more obvious.

Sharing

Let's start with sharing. Even though people typically start with one application, they quickly start deploying more and more. We have customers that have deployed dozens of applications by now! These applications all talk to each other via event streams. Moreover, these applications may dynamically deploy queries to each other. Rapidly integrating new applications is very hard if each application has to link in its own standalone CEP engine.

Scalability

What if the demands on a CEP applications grow, and a single server does not have enough CPU power to satisfy them? If the CEP engine is linked in as a library, there is not much one can do in this situation! On the other hand, a good CEP infrastructure can distribute event processing to multiple servers, balance the load, and satisfy the most stringent scalability demands.

Reliability

What happens if a CEP engine is linked in as a library, and the machine on which it runs dies a sudden death? Well, you've got a problem. You don't have another server standing by ready to take over. You can, of course, replicate both the application and the CEP engine on two machines, but you'll have to figure out how to synchronize the two copies, how to share state, etc. All these problems are already solved by a good server infrastructure product such as Coral8.

But wait...

So why do people even consider linking a CEP engine as a library? There are a few reasons, but they are somewhat misguided:

First, some people don't like the idea of starting a new process for a CEP server. While I sympathize with this sentiment, the price of starting a new process is usually not that high. Complex applications these days have many parts, and always never reside in one process anyway.

The second reason sounds more serious: latency! People think that if their application has to send events to a separate server, and then receive results back, the total processing latency will be a lot higher. Thus, they want to link the CEP server into their application.

This sort of makes sense, except for where do the events come from? In most cases, they don't come from the app, but from some third place: from a financial exchange, a web site, a sensor network, a message bus, etc. The right solution is not to send these events to the application, but directly to the CEP engine! Good CEP engines come with highly optimized adapters, some of them running in the process of the CEP engine itself. These adapters can process incoming events much faster than whatever an application can do! We have seen how some Coral8 adapters can process events in well under 100 microseconds! Why would an application want to re-implement these adapters? It makes little sense.

Finally, the third reason may be more emotional than anything else: Control. Some developers feel that if the engine is "just a library", they have more control over such things as threading model, memory management, and so on. Well, again, why would somebody want to worry about these issues, when CEP vendors spend years and years optimizing the performance of the CEP engines?

The CEP Cloud

I hope I've convinced you that the future of CEP belong to server-based infrastructure rather than CEP libraries. In fact, we have customers today who don't even care about our performance on a single server. They have a vision of running a cloud of CEP servers, and deploying hundreds and thousands of applications on this cloud. Of course, the cloud of these servers has to be completely self-managing. One does not need to worry about which server or servers are running a given app at any given moment, what happens when demand changes, what happens when servers go down, etc. We like this vision a lot. This is our vision, too. You just can't realize this vision via a simple library.

Mark Tsimelzon
President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/einstein-time-and-cep 2007-12-07T10:47:43-08:00 2007-12-10T17:49:06-08:00 Mark

It turns out that handling time correctly inside a CEP engine is a very complex issue, with many implications. Some approaches to time handling in CEP systems are discussed in depth in the 2004 paper by U. Srivastava and J. Widom: Flexible Time Management in Data Stream Systems, but the paper is very technical. There is a much more approachable paper on Coral8's web site written by our very own Bob Hagmann: Fundamentals of Time in Coral8. I am not going to repeat here what these papers say, but rather I want to talk about some practical implications, relevant to anybody using a CEP engine.

Before I go there, however, I want to explain why the whole issue is so intricate. The first reaction of some folks new to CEP, when first exposed to time management issues, is predictable: why are you making this so complex??? I'm sure Einstein has heard the same objection. But in this case, the objection seems quite logical. Most non-CEP applications just don't worry about time too much. Every computer has a system clock, which reliably keeps the time. Every operating system has a number of system calls to access this clock. So what's so hard about this?

The reason this problem is so hard is that CEP applications are typically highly distributed. Events are often generated far away from where they are processed. Think of sensor networks, or of trading applications that subscribe to data feeds from multiple exchange. Even if physically the data source and the CEP engine are not that far from each other, there may be non-trivial latency in getting events in.

So by the time the events from multiple source reach the CEP engine, they may be delayed, typically by different time deltas, and may even be out of order! If you want to analyze precisely what has happened, the system clock has very little relevance to the times when events were generated! So rather than using system clock, the CEP engine must use virtual stream clock, which is the clock driven by the arrival of events on one or more input streams.

If events arrive fast and in-order, this virtual stream clock may just run slightly behind the system clock. But if events are delayed, and have to be synchronized across multiple event sources, and have to be pre-sorted to handle out-of-order issues, then the "virtual stream clock" is quite different from the system clock.

Therefore, the Coral8 Engine provides two modes for analyzing data: one according to the system clock, another according to the virtual stream clock, or how we call this in our documentation, according to event timestamps. And in this latter mode our engine can automatically synchronize and pre-sort events coming from multiple data sources. This takes the deep magic described in the Stanford paper.

If one is analyzing real-time data, virtual stream clock is normally somewhat behind the system clock, due to transport layer delay. There are also use cases where the virtual stream clock runs much faster than the system clock! For example, one of the common use cases on Wall Street is back-testing of trading strategies. Once somebody comes up with a new strategy, they need to test it on historical data.

Playing historical data back to the CEP engine is not a problem, but people typically want to speed up this process as much as possible. So now the time is compressed! Einstein would appreciate some of the issues this causes. What in reality took 1 hour, may take 1 minute in the accelerated playback mode. A jumping 1 hour long window will be emptied every 1 minute, not every 1 hour. Of course, the Coral8 Engine implements the accelerated mode correctly, where no matter how fast you go, the results are guaranteed to be exactly the same.

I've touched upon some of these notions in my post on Determinism in CEP, but I did not quite explain why some CEP engines are deterministic, and some less so. Hopefully now this is a bit clearer. Some engines handle virtual stream time, and some do not. Some handle virtual stream time for one stream, but not for many. Yet handling virtual stream time for multiple streams is the only way to address the issues we have talked about here. It may be non-trivial, but there is no way around it. Like Einstein said, everything should be made as simple as possible but not simpler.

Mark Tsimelzon, President & CTO, Coral8 ]]>

It turns out that handling time correctly inside a CEP engine is a very complex issue, with many implications. Some approaches to time handling in CEP systems are discussed in depth in the 2004 paper by U. Srivastava and J. Widom: Flexible Time Management in Data Stream Systems, but the paper is very technical. There is a much more approachable paper on Coral8's web site written by our very own Bob Hagmann: Fundamentals of Time in Coral8. I am not going to repeat here what these papers say, but rather I want to talk about some practical implications, relevant to anybody using a CEP engine.

Before I go there, however, I want to explain why the whole issue is so intricate. The first reaction of some folks new to CEP, when first exposed to time management issues, is predictable: why are you making this so complex??? I'm sure Einstein has heard the same objection. But in this case, the objection seems quite logical. Most non-CEP applications just don't worry about time too much. Every computer has a system clock, which reliably keeps the time. Every operating system has a number of system calls to access this clock. So what's so hard about this?

The reason this problem is so hard is that CEP applications are typically highly distributed. Events are often generated far away from where they are processed. Think of sensor networks, or of trading applications that subscribe to data feeds from multiple exchange. Even if physically the data source and the CEP engine are not that far from each other, there may be non-trivial latency in getting events in.

So by the time the events from multiple source reach the CEP engine, they may be delayed, typically by different time deltas, and may even be out of order! If you want to analyze precisely what has happened, the system clock has very little relevance to the times when events were generated! So rather than using system clock, the CEP engine must use virtual stream clock, which is the clock driven by the arrival of events on one or more input streams.

If events arrive fast and in-order, this virtual stream clock may just run slightly behind the system clock. But if events are delayed, and have to be synchronized across multiple event sources, and have to be pre-sorted to handle out-of-order issues, then the "virtual stream clock" is quite different from the system clock.

Therefore, the Coral8 Engine provides two modes for analyzing data: one according to the system clock, another according to the virtual stream clock, or how we call this in our documentation, according to event timestamps. And in this latter mode our engine can automatically synchronize and pre-sort events coming from multiple data sources. This takes the deep magic described in the Stanford paper.

If one is analyzing real-time data, virtual stream clock is normally somewhat behind the system clock, due to transport layer delay. There are also use cases where the virtual stream clock runs much faster than the system clock! For example, one of the common use cases on Wall Street is back-testing of trading strategies. Once somebody comes up with a new strategy, they need to test it on historical data.

Playing historical data back to the CEP engine is not a problem, but people typically want to speed up this process as much as possible. So now the time is compressed! Einstein would appreciate some of the issues this causes. What in reality took 1 hour, may take 1 minute in the accelerated playback mode. A jumping 1 hour long window will be emptied every 1 minute, not every 1 hour. Of course, the Coral8 Engine implements the accelerated mode correctly, where no matter how fast you go, the results are guaranteed to be exactly the same.

I've touched upon some of these notions in my post on Determinism in CEP, but I did not quite explain why some CEP engines are deterministic, and some less so. Hopefully now this is a bit clearer. Some engines handle virtual stream time, and some do not. Some handle virtual stream time for one stream, but not for many. Yet handling virtual stream time for multiple streams is the only way to address the issues we have talked about here. It may be non-trivial, but there is no way around it. Like Einstein said, everything should be made as simple as possible but not simpler.

Mark Tsimelzon, President & CTO, Coral8 ]]> http://www.coral8.com/blogs/blog-entry/real-time-view-cep-evaluation-project 2007-11-26T20:51:41-08:00 2007-11-26T20:51:41-08:00 Mark

blog about implementing different use cases using Coral8 and other technologies.

Think about how things are changing! The whole process of evaluating and choosing a technology vendor is still quite opaque. The customer evaluates different products, yet nobody knows who all the candidates are, how the evaluations progress, what each product's challenges are, and how the winner is selected. The selection process is hidden from everyone. If you are lucky, a few months later the chosen vendor would publish a "use case", which will have more marketing than real technical meat.

Contrast this to the evaluation that Marc is doing. Everything is in the open. Competitors get to comment on each other's solutions. If Marc has a problem with some product, the world knows about it the next minute! This spirit of full disclosure may be uncomfortable to some, but we welcome it wholeheartedly. Both customers and vendors suffer when enterprise software is sold behind closed doors.

I guess if other customers start following Marc's lead, we'll soon have to use our very own CEP engine to subscribe to their blogs, to have real-time visibility of what our customers are doing and to respond to issues. We need to stay one step ahead of them. It's a good thing we already have an RSS adapter :)

Mark Tsimelzon
President & CTO, Coral8 ]]>

blog about implementing different use cases using Coral8 and other technologies.

Think about how things are changing! The whole process of evaluating and choosing a technology vendor is still quite opaque. The customer evaluates different products, yet nobody knows who all the candidates are, how the evaluations progress, what each product's challenges are, and how the winner is selected. The selection process is hidden from everyone. If you are lucky, a few months later the chosen vendor would publish a "use case", which will have more marketing than real technical meat.

Contrast this to the evaluation that Marc is doing. Everything is in the open. Competitors get to comment on each other's solutions. If Marc has a problem with some product, the world knows about it the next minute! This spirit of full disclosure may be uncomfortable to some, but we welcome it wholeheartedly. Both customers and vendors suffer when enterprise software is sold behind closed doors.

I guess if other customers start following Marc's lead, we'll soon have to use our very own CEP engine to subscribe to their blogs, to have real-time visibility of what our customers are doing and to respond to issues. We need to stay one step ahead of them. It's a good thing we already have an RSS adapter :)

Mark Tsimelzon
President & CTO, Coral8 ]]>