Geeks With Blogs

News
Charles Young
Microsoft has just released the November CTP (CTP 3) of StreamInsight.   See http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=01c664e4-1c98-4fc8-93ee-08cc039503c1.
I've encountered some confusion from more than one person about the status of StreamInsight, so let me explain briefly that a CTP is a Community Technology Preview. It's not a beta as such. CTPs are closely aligned with the shorter iterative development cycles of agile methodologies.   Many Microsoft product teams use them to provide insight into their progress from a much earlier stage than would be the case for a formal beta programme. So, StreamInsight is a work in progress.   It hasn't shipped yet.   It isn't necessarily 'code complete'.   It doesn't have full documentation.   Bits are missing or may not work.   Under the terms of the license, you can't use it in production.   For some reason, CTPs seem to cause a lot of confusion, even in the Microsoft community, but the clue is mainly in the word 'preview' and partly in the word 'community'.
So what is new?   Well, looking at the help file...
Hopping windows.   This is an additional temporal window model which was conspicuous by its absence in CTP2. It is, very simply, a contiguous 'batched' window and will be very familiar to users of other CEP products.   At any one time the window provides access to the set of events that are alive within that time frame.   After a fixed interval the Window is incremented in time to provide access to events in the next fixed period.   Because events in StreamInsight are interval-based, an event may appear in more than one time frame.   Note that the graphic in the help file appears to be slightly incorrect. Event e1 is unaccountably extended in the first two 'hops'.
Update: After watching Roman Schindlauer's PDC '09 session and re-reading the help file with a little more care, I realised I had not fully understood this new model.   This window model is defined by two values.   The window has a fixed length, 'S', but also a hop size, 'H'.  At each hop, a new window is created of size S.  
If H == S (and the two intervals start and end at the same time), you get a special case of a hopping window which Microsoft calls a 'tumbling' window - i.e., a series of contiguous, non-overlapped windows as I described above.   If H < S, each window can overlap other windows.
So, my apologies.  I didn't properly understand this new model to begin with.  However, this doesn't change the observation that the help file graphic needs correcting.   Also, the current help file does not provide a comprehensive explanation.   The exact behaviour of 'clipping' isn't defined (what about events that appear in multiple windows?), and there is no mention of what happens if you set H > S (which may, of course, not be allowed).
Joins.   Again, highly conspicuous by its absence in CTP 2, CTP 3 now supports joins in LINQ!   The fact that we have only just got this fundamental capability (what could be more central to CEP than the ability to specify joins directly in your query language)  shows just how early a view Microsoft gave us in CTP 2 (CTP 1, incidentally, was a private preview released earlier this year to company employees only).
User-Defined Aggregates (UDAs).   This feature allows developers to create highly customised aggregation operators for aggregating over windows. Developers create UDAs by deriving from one of two classes depending on the need to access timestamp information and also following a reasonably straightforward pattern involving creation of an annotated extension method that allows the UDA to be accessed directly within LINQ.
User-Defined Operators (UDOs).   UDAs map multiple events to a single scalar value. Like UDAs, UDOs take multiple events as input but can return multiple events.   They follow a very similar design pattern to UDAs, but return an IEnumerable<T> rather than a single value.
UDAs and UDOs provide a major piece of the CEP jigsaw and greatly extend the power and expressivity offered by LINQ.
Additional Query Improvements.   TopK queries now allow lambdas to be used in Take() to project values from ranked events into the payload of each of the selected events. The Group and Apply operators have also been improved.
In CTP 2, .NET methods in expressions included in LINQ were evaluated at query translation time, which wasn't much use.   In CTP 3 they are now evaluated at run time.
CTI improvements. In CTP 2, CTIs could only be enqueued in code in an input adapter.   CTP 3 provides support for declarative CTI generation by allowing query template bindings to be configured with an 'AdvanceTime' specifier. You can also configure an adapter factory. CTIs are then generated internally by the engine.   CTIs are very cool.   Declarative CTIs are very, very cool :-)
Management Service API Improvements.   A number of improvements have been made to the management service API to support a richer set of views at different levels within the engine.
So, in summary, Microsoft has made significant progress since CTP2, and CTP 3, as we might hope, looks more like a complete engine.   Major gaps in functionality have been filled and we can begin to see more clearly what capabilities will be included in the first release.   Keep up the good work

 

Posted on Tuesday, November 17, 2009 4:12 PM | Back to top


Comments on this post: Microsoft StreamInsight CTP 3 - New Features

No comments posted yet.
Your comment:
 (will show your gravatar)


Copyright © Charles Young | Powered by: GeeksWithBlogs.net