As nice as Continuous Queries are, it turns out that sometimes CEP users want to issue the traditional old-style Snapshot queries. Why would they want to do that? Many reasons, really. Some of them want to integrate with traditional BI tools that cannot output results of Continuous Queries. Some really want to get the one time snapshot of the data. Some simply want to query the state kept in the memory of their CEP engine, in so called "windows".
To solve this problem, Coral8 added a while back the notion of "public windows", which can be queried using plain old standard SQL in a snapshot fashion. Problem solved? Not so fast...
A financial institution may use a CEP engine to track the list of all order for a day in a window. At any given time, one may want to ask, how many orders in the incomplete state are there, grouped by trader, security, or exchange? So far it sounds like a snapshot query, right? But once the result is returned, the user typically wants to keep track of the results of this query! As more orders are added and existing orders change their state, the counts returned by the query are changing. People want their CEP engine to track these changes.
Consider an even simpler example from the RFID tags. Let's say that you are tracking the list of all assets with RFID tags in a Coral8 window. Imagine you want to query the window for the list of all tags that have not sent an update to a reader in the past 10 seconds. It's a useful list, but it can be even more useful if it's dynamically updated as the underlying window changes!
One challenge with building this feature was that a standard non-public Coral8 window is only visible from the module where it's declared. Public windows are visible from the outside, but they only allow snapshot queries via SQL, and there are no live updates. So to implement Snapshot Queries with Live Updates, we added a new and powerful feature: Shared Windows.
To share and query a window across modules, one sets up a Master Window, which keeps track of all the relevant data (orders, RFID tags, etc.). Once a Master Window is set up, a Mirror Window can be created in a separate module, and it can dynamically get a subset of data from the Master window. Once this initial subset is received, the Mirror data starts receiving updates on all additions and deletions from the Master window. Thus, a continuous query on the mirror window first returns the snapshot, followed by live updates. Voila!
Of course, the beauty of this whole set up is that a mirror window and the query over the mirror window can be set up completely dynamically. Coral8 has always been known for its ability to register queries dynamically, so this is probably not too surprising. In our testing, a mirror window with a new query can be created, compiled, registered, and relied upon to produce a result in well under 1 second! I personally find this quite amazing.
Mark Tsimelzon, President & CTO, Coral8 ]]>
As nice as Continuous Queries are, it turns out that sometimes CEP users want to issue the traditional old-style Snapshot queries. Why would they want to do that? Many reasons, really. Some of them want to integrate with traditional BI tools that cannot output results of Continuous Queries. Some really want to get the one time snapshot of the data. Some simply want to query the state kept in the memory of their CEP engine, in so called "windows".
To solve this problem, Coral8 added a while back the notion of "public windows", which can be queried using plain old standard SQL in a snapshot fashion. Problem solved? Not so fast...
A financial institution may use a CEP engine to track the list of all order for a day in a window. At any given time, one may want to ask, how many orders in the incomplete state are there, grouped by trader, security, or exchange? So far it sounds like a snapshot query, right? But once the result is returned, the user typically wants to keep track of the results of this query! As more orders are added and existing orders change their state, the counts returned by the query are changing. People want their CEP engine to track these changes.
Consider an even simpler example from the RFID tags. Let's say that you are tracking the list of all assets with RFID tags in a Coral8 window. Imagine you want to query the window for the list of all tags that have not sent an update to a reader in the past 10 seconds. It's a useful list, but it can be even more useful if it's dynamically updated as the underlying window changes!
One challenge with building this feature was that a standard non-public Coral8 window is only visible from the module where it's declared. Public windows are visible from the outside, but they only allow snapshot queries via SQL, and there are no live updates. So to implement Snapshot Queries with Live Updates, we added a new and powerful feature: Shared Windows.
To share and query a window across modules, one sets up a Master Window, which keeps track of all the relevant data (orders, RFID tags, etc.). Once a Master Window is set up, a Mirror Window can be created in a separate module, and it can dynamically get a subset of data from the Master window. Once this initial subset is received, the Mirror data starts receiving updates on all additions and deletions from the Master window. Thus, a continuous query on the mirror window first returns the snapshot, followed by live updates. Voila!
Of course, the beauty of this whole set up is that a mirror window and the query over the mirror window can be set up completely dynamically. Coral8 has always been known for its ability to register queries dynamically, so this is probably not too surprising. In our testing, a mirror window with a new query can be created, compiled, registered, and relied upon to produce a result in well under 1 second! I personally find this quite amazing.
Mark Tsimelzon, President & CTO, Coral8 ]]>
I've been trying to understand why this is easy for some customers and hard for others, and I've recently observed a pattern I want to share. I'm sure it is controversial and does not explain the whole story, but still. Here it goes:
Those people who find it hard to come up with specific use cases think about The Event Cloud. Those who find it easy to do think in term of event streams.
Now, if you were following the recent "CEP vs. ESP" wars, you may be forgiven for thinking that the basic difference between clouds and streams is about ordering. Streams carry ordered events and clouds don't require ordering, or so the story goes. Well, we all know this is false. Coral8 Engine, as well as some other ESP/CEP engines can handle unordered streams. So ordering is a red herring.
The real difference between a cloud and a stream is much more significant. An event cloud is an abstract concept. In some sense, it does not really exist! An event cloud represents a collection of all the events flowing through an enterprise or a part of an enterprise. It may be useful to think about this collection from a theoretical standpoint, but in my experience, it does not help one come up with and understand use cases.
Streams, on the other hand, are much more real and useful. Each stream has a meaning and carries similar events. You can decompose your events into input streams, output streams, and intermediate streams. Each stream (at least in the Coral8 Engine) has a name (a URI) and is a publish/subscribe topic, so you may have multiple subscribes and publishers dynamically added. Also, each stream may have an access control list associated with it.
Once you think about a stream, you can think about the things you may want to do with it, such as filter events, maintain a window, aggregate events over a window, enrich the stream, join it with another stream, persist it to a database, and so on. Your use cases start practically defining themselves as soon as you decompose your problem into streams and operations on streams.
Hopefully this has convinced you to stop thinking about "The Cloud" and start thinking about multiple streams, at least for the purpose of defining use cases. As some folks know, I like finding non-CEP analogies for CEP topics. Here is one here: You'll never understand databases if you think that they just store "data". To understand databases, you've got to understand that they store tables. Similarly, to uncloud and streamline your thinking about CEP, start thinking streams.
Mark Tsimelzon, President & CTO, Coral8. ]]>
I've been trying to understand why this is easy for some customers and hard for others, and I've recently observed a pattern I want to share. I'm sure it is controversial and does not explain the whole story, but still. Here it goes:
Those people who find it hard to come up with specific use cases think about The Event Cloud. Those who find it easy to do think in term of event streams.
Now, if you were following the recent "CEP vs. ESP" wars, you may be forgiven for thinking that the basic difference between clouds and streams is about ordering. Streams carry ordered events and clouds don't require ordering, or so the story goes. Well, we all know this is false. Coral8 Engine, as well as some other ESP/CEP engines can handle unordered streams. So ordering is a red herring.
The real difference between a cloud and a stream is much more significant. An event cloud is an abstract concept. In some sense, it does not really exist! An event cloud represents a collection of all the events flowing through an enterprise or a part of an enterprise. It may be useful to think about this collection from a theoretical standpoint, but in my experience, it does not help one come up with and understand use cases.
Streams, on the other hand, are much more real and useful. Each stream has a meaning and carries similar events. You can decompose your events into input streams, output streams, and intermediate streams. Each stream (at least in the Coral8 Engine) has a name (a URI) and is a publish/subscribe topic, so you may have multiple subscribes and publishers dynamically added. Also, each stream may have an access control list associated with it.
Once you think about a stream, you can think about the things you may want to do with it, such as filter events, maintain a window, aggregate events over a window, enrich the stream, join it with another stream, persist it to a database, and so on. Your use cases start practically defining themselves as soon as you decompose your problem into streams and operations on streams.
Hopefully this has convinced you to stop thinking about "The Cloud" and start thinking about multiple streams, at least for the purpose of defining use cases. As some folks know, I like finding non-CEP analogies for CEP topics. Here is one here: You'll never understand databases if you think that they just store "data". To understand databases, you've got to understand that they store tables. Similarly, to uncloud and streamline your thinking about CEP, start thinking streams.
Mark Tsimelzon, President & CTO, Coral8. ]]>
Coral8 is one of 6 finalists in the “Database Engines and Data Tools” category. It is especially gratifying that the Coral8 Engine is a finalist because the category is much broader than Complex Event Processing, showing the high market interest in CEP and Coral8 within the general information management category.
The Jolt Award committee takes a deep, serious look at the products nominated for their Product Excellence Awards, even downloading and evaluating the products directly. The fact that Coral8 is a finalist shows the richness of the feature set and the completeness of the product for developers to build and deploy CEP applications.
It is also nice to note that Coral8 is the only CEP product that is a finalist for a Jolt Award.
Thank you to our customers who helped make our product so popular, and many thanks to our excellent product development team who has created such a fantastic product.
To read more about the Jolt Product Excellence Awards, please click here. To see the finalists across all categories, please click here.
Thanks for reading,
John Morrell
Coral8 is one of 6 finalists in the “Database Engines and Data Tools” category. It is especially gratifying that the Coral8 Engine is a finalist because the category is much broader than Complex Event Processing, showing the high market interest in CEP and Coral8 within the general information management category.
The Jolt Award committee takes a deep, serious look at the products nominated for their Product Excellence Awards, even downloading and evaluating the products directly. The fact that Coral8 is a finalist shows the richness of the feature set and the completeness of the product for developers to build and deploy CEP applications.
It is also nice to note that Coral8 is the only CEP product that is a finalist for a Jolt Award.
Thank you to our customers who helped make our product so popular, and many thanks to our excellent product development team who has created such a fantastic product.
To read more about the Jolt Product Excellence Awards, please click here. To see the finalists across all categories, please click here.
Thanks for reading,
John Morrell