Last week we announced the release of version 5.5 of Coral8 Engine, which has a number of very cool features. The biggest and the most exciting of them must surely be Snapshot Queries with Live Updates. Here is why:

History lesson: Continuous and Snapshot Queries

First generation CEP engines started with the concept of Continuous Queries. Unlike database queries that return a snapshot of data at a given time, Continuous Queries are registered once, subscribe to the relevant input streams, and then continuously produce streaming output on one or more output streams. Very nice. Continuous Queries are the foundation of all CEP engines today.

As nice as Continuous Queries are, it turns out that sometimes CEP users want to issue the traditional old-style Snapshot queries. Why would they want to do that? Many reasons, really. Some of them want to integrate with traditional BI tools that cannot output results of Continuous Queries. Some really want to get the one time snapshot of the data. Some simply want to query the state kept in the memory of their CEP engine, in so called "windows".

To solve this problem, Coral8 added a while back the notion of "public windows", which can be queried using plain old standard SQL in a snapshot fashion. Problem solved? Not so fast...

Snapshot Queries with Live Updates: Why?

It turns out that what many customers really want is a combination of Snapshot and Continuous Queries. They want to get the snapshot result of the query, followed by a live stream of updates to the result of the query! Why do they want it? Let's look at a couple of examples:

A financial institution may use a CEP engine to track the list of all order for a day in a window. At any given time, one may want to ask, how many orders in the incomplete state are there, grouped by trader, security, or exchange? So far it sounds like a snapshot query, right? But once the result is returned, the user typically wants to keep track of the results of this query! As more orders are added and existing orders change their state, the counts returned by the query are changing. People want their CEP engine to track these changes.

Consider an even simpler example from the RFID tags. Let's say that you are tracking the list of all assets with RFID tags in a Coral8 window. Imagine you want to query the window for the list of all tags that have not sent an update to a reader in the past 10 seconds. It's a useful list, but it can be even more useful if it's dynamically updated as the underlying window changes!

Snapshot Queries with Live Updates: How?

Hopefully these two simple example convince you that Snapshot Queries with Live Updates are very useful in a number of settings. So how do they work?

One challenge with building this feature was that a standard non-public Coral8 window is only visible from the module where it's declared. Public windows are visible from the outside, but they only allow snapshot queries via SQL, and there are no live updates. So to implement Snapshot Queries with Live Updates, we added a new and powerful feature: Shared Windows.

To share and query a window across modules, one sets up a Master Window, which keeps track of all the relevant data (orders, RFID tags, etc.). Once a Master Window is set up, a Mirror Window can be created in a separate module, and it can dynamically get a subset of data from the Master window. Once this initial subset is received, the Mirror data starts receiving updates on all additions and deletions from the Master window. Thus, a continuous query on the mirror window first returns the snapshot, followed by live updates. Voila!

Of course, the beauty of this whole set up is that a mirror window and the query over the mirror window can be set up completely dynamically. Coral8 has always been known for its ability to register queries dynamically, so this is probably not too surprising. In our testing, a mirror window with a new query can be created, compiled, registered, and relied upon to produce a result in well under 1 second! I personally find this quite amazing.

Want to learn more?

Of course, this blog post has just a very short intro to the magical world of Shared Windows and Snapshot Queries with Live Updates. To learn more, you can refer to our documentation (search for Shared Windows), or better yet, read an excellent white paper by Bob Hagmann on this very subject. It covers how Snapshot Queries with Live Updates work, how they can be created in CCL, how they can be registered dynamically from Java, and lots of other relevant topics.

Mark Tsimelzon, President & CTO, Coral8