Event processing - small data vs. big data and the Sorites Paradox.

This picture is taken from a blog post from the "Big Data Journal" by Jim Kaskade entitled "Real-time Big Data or Small Data".  

Kaskade attempts to define quantitative metrics to what is "small data" vs. what is "big data".  

In terms of throughput big data is defined as >> 1K event per second, while small data is << 1K per second, I guess that around 1K event per second is defined as medium data...  

On variety big data is defined as at least 6 sources of structured events and at least 6 sources of unstructured events.  There are other dimensions like - small data relates to one function in the organization, while big data to several lines of business.     

The attempt to define where "big data" starts is interesting, the main issue is what are the conditions in which implementation of systems should become different, and here the borders are not that clear, since there are currently systems that can scale both up and down.

Interestingly -- "Big" and "Small" are fuzzy terms.  Which reminds me on one of the variations of the Sorites Paradox,  that I've came across during my Philosophy studies, many years ago, which goes roughly like this.

Claim:  Every heap of stones is a small heap.

Proof by mathenatical induction.

Base:  A heap of 1 stone is a small heap

Inductive step:  Take a small heap of K stones and add 1 stone, surely it will stay a small heap.

SAP acquired Sybase: the event processing angle

It seems that Paul Vincent will have to update his product genealogy again. There are various analysts like James Kobielus or Philip Howard have expressed their opinion about the motivation and possible implications of this merge, and I leave this discussion to them; I'll write on the narrow perspective of the event processing angle (to justify the name of this Blog).

Taking the EP perspective, there is a series of big fish swallows smaller fish, until it gets to the ultimate big fish. Event processing is shifting from being dominant by start-ups to be dominant by bigger companies, which happened before in other areas, and is sign of getting this technology to the main stream of computing, this is consistent with some of the trends we identified as the current trends in the conclusion chapter of the EPIA book, event processing is going from narrow domains to wider domains and from being stand-alone technology to being part of bigger frameworks. As four big fish -- now the four big software companies - IBM, Microsoft, Oracle and SAP are all in the EP market, and all of them view EP as part of a bigger game; I guess that there will always be smaller stand-alone EP companies for niche markets, but most applications do not live in an isolation and have some relations to other areas - BPM, BRMS, BI and some more.

The fact that most players in this market are big and medium (TIBCO, Progress Software, Software AG) companies, can make the climate more comfortable to advance standards now -- which is one of the topics we plan to discuss in Dagstuhl next week.

When we established EPTS, SAP was not interested to join, since they did not have a product at that time, we'll see whether they will want to inherit Sybase's membership (which by itself was recently inherited from Aleri)... We'll stay tuned for more on this front...

Some footnotes to Brenda Michelson's post on TUCON

Las Vegas is a place where companies like to do customer conferences; I guess that the quantity and sizes of hotels is a major factor. Last week IBM has done the big customer conference: IMPACT 2010, after a few years of attending this conference, I have skipped this one, so I did not cover it in the Blog. This week TIBCO has done its customer conference - TUCON 2010, naturally I have not been there either, as I am not a TIBCO user, but looked at some impressions on the Blogland, and came across Brenda Michelson's Blog describing the keynote sessions. With the disclaimer that my information on this is second hand (but a good hand !), I'll make some footnotes:

1. I believe in the direction that events will be more and more in the center of enterprise (and other applications); we see it happening and we'll see much more.

2. Like Brenda, I am not sure that the phrase "enterprise 3.0" will catch, at the past there was an attempt to push "SOA 2.0" but it did not catch either, I think the world became tired from recycling the n.0 phrase.

3. "2 second advantage" is catchy, and may be true for some applications, but in some of them 2 second might not be enough to make a difference. In general there are two aspects: immediate reaction -- react fast (or faster than others); and proactive --- react in time to mitigate or eliminate future event; I am not sure it that if the Interpol will know about a crime 2 seconds earlier it will be enough time to eliminate the crime.

4. About NoSQL and relations to event processing - I'll write a separate post, it is an interesting insight that NoSQL is gaining traction.

5. Context as additional dimension -- I have written several times in the past about contexts, and am planned to deliver a tutorial about contexts in DEBS 2010; I believe that contexts are starting to be a major factor in computing.

It is good to see that the event-centric universe is getting more traction. More -later.

On small vendors, big vendors - individual view and industry-wide view

If you want to have a pet animal, you might prefer the little kitten over the big dog; if yo want to have somebody to guard your house, a big dog might be more effective, there also people who prefer big dogs as pets, a matter of personal taste.

Marc Adler responded to my previous post about consolidation and pure play in the EP market. Since Marc is in blog reduction mode, I will not be insulted if he'll not react to this posting, but I would like to provide further perspective.

There is nothing in Marc's response that I don't agree with, in fact when I have been 30 years younger (having 30 kg less from today, and this is after I've reduced 21 kg in the recent year).

I have worked in the IT shop of the Israeli Air-Force, I have been working with a new product that was called DB/1 and later renamed as Sapiens, at that time positioned as application generator for data-driven programming (there has been evolution in positioning over the years as well), it was a product in its beginning, I have done the first project with this product on the premises of the product developers, went with them to lunches, and went to their family weddings and funerals, thus, I got excellent service for them, could ask to add features, when there was any bug or problem I know exactly who is the responsible developer and could call him, or even invite him to solve it in our site. This has worked well since I had good personal contacts with them, but moreover, at that time the number of customers they had could be counted on the fingers of a single hand, and they could provide much attention to every customer. As said, for me, it was the ideal product to work with, and I fought some of my colleagues and superiors who thought that this is a too big risk for the Air-Force to depend on them in critical application, which was of course true. I guess that Marc had somewhat similar experience with Coral8 in his previous work; I also feel sympathy to the claim that big corporates have an inclination to come to customers with more people than the customer expects to see, and has less intimate atmosphere with customers, this was always been true.

Fast forward 30 years, I have somewhat different perspective on the universe; it was somehow surprising for me to find myself being hired by a big corporate, from an employee's point of view there are pros and cons to be employed by a big corporate, there is a nice posting on this topic.

People whose small companies are acquired by big corporates sometime dislike the culture of big corporates and move on, sometime they adjust, I know stories of both types, it is not black or white.

My perspective now is more on the macro level and looking at the question of: Is the current wave of consolidation good or bad in general for the event processing area, its assimilation into main-stream computing, and the ability of play a significant role in current and future enterprise computing?

My own view is that the market moves to the right direction. As I believe that the larger market is not in stand-alone event processing applications, but as a pervasive technology embedded in enterprise computing in general, there is some benefit to companies like - IBM, Oracle, TIBCO, Sybase, Software AG, and now we also see that Progress Software is making event processing as part of a more general platform. Some applications may not need it, but many others do, and getting event processing in the mainstream enterprise software infrastructure, can be done through owners of such existing infrastructure.

One other potential benefit that I see is that with a market that is more dominated by bigger vendors there is stronger probability to get to standards. Standards is one of the signs of maturity for an area (e.g databases, web services), and will be vital to get event processing into the mainstream. We started to discuss standards in the pre-EPTS meetings in 2006, and at that time the dominant startup companies were very much against it, since they both did not see the value for themselves, and also feared that standards will distract their limited resources. Bigger companies have more standard oriented culture, and experience in other areas of how to do it right. I think that the current developments in the market provides a good opportunity to raise the standards issue again, and this will be one of the topics planned to be discussed in the upcoming Dagstuhl seminar on event processing. I'll write more about standards follows the conclusion of the Dagstuhl seminar.

Back to the original theme --- there are times in life in which one prefers small cats, and other times that one prefers big dogs. Small cats may be more cute and pleasant, but in this phase in my life I am going to hunt, and big dog will probably be more effective for that task.

On event processing building blocks

As response to my posting about stand alone or part of a bigger picture, Marco made a comment that in the event-driven world there is an opportunity to provide building blocks that may be used in various platforms and part of multiple bigger pictures. This is a valid point, as event processing is indeed a collection of capabilities of various types -- transformation, aggregations, pattern detection and more, each of them can also be of various types --- e.g. supporting variety of aggregating functions, variety of transformation, multiple patterns etc, coupled with the fact that event-driven computing is decoupled, thus the interfaces to these components can be quite simple --- receive events and send events, can provide an opportunity to provide variety of such components, kind of plug-and-play event processing. The key, and currently main obstacle in letting it happen is standardization and current lack of standards.

• Interoperability standards are needed for standard interfaces
• Event meta-data standards are needed so that the events exchanged between components
• Languages and meta-modeling standards so people can model and program in a seamless model.

I believe that this is the right direction going forward, and it is a direction we in Haifa Research Lab are investigating; this might be the key to make event processing a pervasive technology - more about it, later.

Event processing - stand-alone, part of a bigger game, or both?

Following my previous posting, somebody told me that I was vague about whether I think that event processing as stand-alone technology is good or bad. Well -- when I was a student I took "image processing" course, and we worked on a graphical screens with 256 gray levels, since then I realized that even in "black and white" picture there are a lot of gray area Thus, I cannot classify it as "good" or "bad". But I'll provide some observations:

1. The event processing area started as "stand alone" engines, this has been obvious, since it started by start-ups and not by big companies as part of other frameworks
2. There is a gradual shift in the market from stand alone event processing solutions to event processing capabilities embedded in larger frameworks, when bigger companies got into the picture, and this trend has intensified.
3. "Stand alone" products may have to implement functions that "embedded" products can use existing functions in the original framework, such as: routing, transformation, filtering...
4. Unlike some software components that may need tight integration, event processing work in loose coupling relative to other components -- sending and receiving events --- thus this supports the possibility for having stand-alone EP.
5. However, there are no interoperability standards which requires to provide adapters for each producer and consumer, which makes stand-alone EP more difficult, relative to a single framework -- the level of difficulty is a functions of the quantity and diversity of producers and consumers. Enterprise integration framework may already include variety of adapters that the embedded EP can get "for free".
6. Event processing is in many cases part of a bigger application, and in this case, there is a benefit of having a single programming model for the bigger application, and not using different programming models/languages/user interfaces for the various part of the system, this also goes against stand-alone EP; In cases where the system is pure EP, this consideration may not be valid.
7. Stand-alone EP may support heterogeneous components -- e.g. work with DBMS from one vendor, messaging system from another vendor, and connect to BPM system from third vendor, while embedded EP is typically homogeneous, since it all comes from one vendor. This may be true, though today there are a lot of cross-adapters among various components that enable framework to support other components (say DBMS) from other vendors, especially where there are standard interfaces.

Is there a bottom line here? --- I guess that the gray area is that there is some segment of the market in which stand-alone EP can live, but I also think that the trend of moving from stand-alone to embedded will continue to exist.