Driving while looking at the rear-view mirror

Typically I don't recommend commercial Blogs, but I'll make an exception this time, and cite Mark Palmer's post, since I like the metaphor he made, reflected in this picture.  According to Mark, analytics that refers to the past is like driving by only looking at the rear-view mirror,  which of course can show the road you have already passed.  Since typically we drive forward and not backward it should be more useful to look ahead than to look back.   In many cases the road is fixed, in the sense that the road forward looks exactly like the road backward, and then it might make sense to do it,  however, in other cases, like driving in real traffic and not in a bubble, the road ahead may contain surprises that are not evident from the previous parts of the road. 

The interesting thing is that even law latency event processing system are, in fact, looking at the past,  where the past is almost the present,  looking to the future is not a standard event processing feature.   

Some catching up

Have not written for the last week,  spent some of it in the IBM La Gaude site, near Nice, in South France, the La Gaude site is on mountain and can be seen in a bird eye's view here.    The meeting was an internal meeting of "technical leaders" (whatever it is) in IBM South and West Europe, and was kind of educational and networking session.    Back home - occupied by some family matters, so now I found some time to catch up.    First looking at the Blogland,  one can find two old bloggers returning after long silence, one of them is Brian Connell,  who returned to blogging after a long time, the other is Marco from Rulecore, who also returned to blogging after a long silence.  In a comment to my Blog,  the Rulecore guys are wondering why Rulecore is not mentioned in the ACM Computing Survey upcoming survey on event processing while other are mentioned.   I think that the reason is quite obvious, while I am familiar with Rulecore (they participated in the implementation of the EPIA example; my students used them in project several times),  if one wants to be known by the academic community, then one has to participate in the research oriented conferences, publish (at least industrial) papers etc --- in our world, visibility is the name of the game, so you might consider sending an industrial paper or experience report to DEBS 2011,  or participate in the DEBS 2011 challenge.    


Another blog (Jason Irwin's)  that I don't typically follow provides a new review of the EPIA book.     


My next trip is planned for late March in the USA,  I was invited to give a tutorial on event processing as one of the highlights of the OMG Technical Conference,   I might also do some other meetings in the USA (customers' meeting, IBM meetings, university visits),  but still working on the plan.     

Some Blog statistics - December 2010

This year I have not posted the annual statistics about this Blog readership, so taking advantage of the vacation to do it, along with going to movies, musical on stage, and bowling with my daughters.   The Blog now showing on the bottom some of the most popular postings, however, it started recording only several months ago, thus the more accurate statistics are accumulated in Google  Analytics, where this blog is tracked from September 2007.   Starting with the quantities:  There are around 1800 regular readers that are reading each posting in this blog, and additional 3600 who get into the blog from time to time (once every two-three weeks),  There are also  people who entered the blog less frequently, some of them only one time, the total number of this blog visitors is around 65000 people.  The geographic distribution is also interesting, the map is mostly painted in several green variants, so what stands out are the white space, countries from which there was no reader so far.   Europe has not white spot, America has one white spot: Suriname.   Asia has three white spots:  North Korea, Turkmenistan and Tajikistan.   Africa still has only partial coverage, all countries in the north part and south part of the continent are green, but the middle is mostly white.    This may be an indication that the Internet infrastructure in these countries still needs to go some way, or that the content of my blog does not appeal to people in these countries.

As far as the  ten countries with most views, these are:  1). USA; 2). UK; 3). Germany; 4). Canada; 5). Israel; 6). India; 7). France;  8). Japan;  9). Sweden and 10). Australia.  The readers come from 178 countries.

In a city view ten cities with most views, there are: 1). London; 2). NYC; 3). Paris; 4). Karlsruhe;  5). Haifa; 6). Tokyo; 7). Singapore; 8). Bangalore; 9). Göteborg; 10). Vienna. 

About 10.5% of the page views were direct requests, around 25% results of searches, and the rest, references by various websites. 

The most popular posting, is still ,by far, the one entitled "On unicorn, professor and elephant",  which answers a claim that everything done until today in the event processing area is just a hype and worth nothing. Since the time it was written two years ago, there were many proof points the the EP area has value to customers in various industries, and the assertion that it is still an infant, and some vendors do over-hype it is also still valid. 

The second most popular posting, is the one entitled: "On simple event and simple event processing".   This is an early posting.  In the past we used a terminology of: simple event processing (filtering and routing), mediated event processing (aggregation, transformation, composition)  and complex event processing (pattern matching).  However, I stopped using these terms since it got people more confused, due to the fact that different people have different associations with the terms simple and complex, especially the ambiguousness of  complex event processing,  that is interpreted by some as (complex event) processing and by some as complex (event processing).   I also tend to use composite event instead of complex event when talking about event that is composed of events.

The third most popular posting, is the one entitled: "On Enterprise Service Bus and Event Processing"   which is also an early posting,  this also states that event processing capabilities should be part of enterprise computing infrastructure, where ESB is a natural place to be a center point for it.  Since that time EP capabilities became even more pervasive among various technologies.

Somehow related to this is the most popular among the 2010 postings entitled: "Consolidation and pure play in the EP market".  This deals with the fact that most of the EP vendors today are big software vendors that consolidated EP within their products, while there is still a niche for pure play vendors.

While, as the blog title indicates, most of the blog postings deal with event processing, there are several off-topic postings that won a lot of responses, such as the one on positive thinking, and the one in which I described things that I heard from my father about the holocaust.   My last posting on accountability belongs to this family.

This year I spent less time on blogging, so the quantity of postings is less than either 2008 or 2009,  but I intend to catch up.    

The book "Event Processing in Action" is in someway descendant on this blog,  the publishers read the blog before approaching me to write the book,  but of course, there is more emphasis on rigor and quality within the book, the blog is "quick and dirty".      

End of summary -- next posting will go back to professional stuff

Some footnotes to Brenda Michelson's post on TUCON

Las Vegas is a place where companies like to do customer conferences; I guess that the quantity and sizes of hotels is a major factor. Last week IBM has done the big customer conference: IMPACT 2010, after a few years of attending this conference, I have skipped this one, so I did not cover it in the Blog. This week TIBCO has done its customer conference - TUCON 2010, naturally I have not been there either, as I am not a TIBCO user, but looked at some impressions on the Blogland, and came across Brenda Michelson's Blog describing the keynote sessions. With the disclaimer that my information on this is second hand (but a good hand !), I'll make some footnotes:

1. I believe in the direction that events will be more and more in the center of enterprise (and other applications); we see it happening and we'll see much more.

2. Like Brenda, I am not sure that the phrase "enterprise 3.0" will catch, at the past there was an attempt to push "SOA 2.0" but it did not catch either, I think the world became tired from recycling the n.0 phrase.

3. "2 second advantage" is catchy, and may be true for some applications, but in some of them 2 second might not be enough to make a difference. In general there are two aspects: immediate reaction -- react fast (or faster than others); and proactive --- react in time to mitigate or eliminate future event; I am not sure it that if the Interpol will know about a crime 2 seconds earlier it will be enough time to eliminate the crime.

4. About NoSQL and relations to event processing - I'll write a separate post, it is an interesting insight that NoSQL is gaining traction.

5. Context as additional dimension -- I have written several times in the past about contexts, and am planned to deliver a tutorial about contexts in DEBS 2010; I believe that contexts are starting to be a major factor in computing.

It is good to see that the event-centric universe is getting more traction. More -later.

More on responsive, reactive and proactive computing

Earlier this week, I've posted in this Blog short definitions of the terms: responsive computing, reactive computing and proactive computing. Somehow, terminology always gets questions and responses, and there has been a thread in the Complex Event Processing forum that started to discuss it. So as a follow-up to this discussion, some clarification. The computing modes (responsive, reactive, proactive) are indeed mutually exclusive, however a single system may have combination of all of them.

Active databases are example that start with responsive computing, the basic operations are: insert, modify, delete or retrieve from a database. Then the active database engine takes these database operations as events, and apply reactive computing to execute rules reacting to these events.

The opposite direction is a reactive system, doing some kind of event processing, which during the event processing operations need to consult a database in order to enrich an event with more information. The database query issued is a responsive computing.

Proactive computing may also be combined with reactive and responsive systems.

Hans Gilde has posted on the complex event processing forum an example that combines all three.

While responsive computing is the bread and butter of computing, and reactive is now being more understood, proactive is still lagging behind in terms of realizing the potential, maybe the next hill to climb.

On event processing grand challenges

This is the logo of the "Grand Challenges SIG" of the UK Electronic design knowledge transfer network. In the 4th Event Processing Symposium last year, Arkady Godin from MITRE, proposed in the business meeting to launch a work group that will deal with grand challenges in event processing. The EPTS steering committee decided that such a workgroup may be premature, and it is better to complete some work in the other workgroups -- languages, architecture, use cases, interoprability, in order to have a better understanding of the grand challenges, however, this issue has neither forgotten nor forsaken. We determined that the best forum to discuss the grand challenges is the (second) Dagstuhl seminar on event processing, which I'll co-organized with Rainer von Ammon and Mani Chandy, we also asked to invite all people that we saw as the right set of people to participate in such an event (if you have not been invited and think you would like to contribute, please let me know, there is a waiting list for available slots if any, no promises though).

In the coming 5th event processing symposium in Trento, we shall hold the first brainstorming about grand challenges. Pending technical feasibility (still requires confirmation from local organizers) we plan to do this as a public session that all EPTS members will be able to use through audio conference call, EPTS members will get full details with call-in numbers when it will be finalized.

Grand challenges can be in multiple areas and also refer to multiple addressees. There are grand challenges that will require a community effort, and these challenges should be picked up by EPTS, some challenges are to advance the state-of-the-art, and this will be addressed to the research community, with some creative incentives. Some will be referred to the product vendors, and may be some to other adjacent communities.

Idea for such grand challenges are solicited from various sources:

• EPTS workgroups that already are active, each workgroup will be asked to contribute a challenge in its area
• The reach-out sessions for other communities in the symposium (the BPM community and the the IT management community) might also lead to some challenges
• The research community is always a source for such challenges
• Last but not least -- all EPTS members (and other interested people that wish to contribute, and thus are hopefully future EPTS members), specifically customers who has long-term vision about their systems, like our MITRE colleagues who started this discussion, and our analysts colleagues who have cross-vendor and cross-customer perspective.
  
  Anybody who wishes to contribute ideas to this session, please let me know by the end of this week.

I'll wrap-up this posting, by pointing out a Blog entry I have seen just now, written by Niels, who runs a consultancy company called "SQL Develop" as its name testifies deals with DB related issues. The upcoming Microsoft product "StreamInsight" has raised the interest in the database community, as Microsoft seems to take an SQL oriented approach to its product, and Niels is blogging about "CEP resources" he covers Blogs in this area, on my Blog he writes: THE blog about CEP, if you were to read only one blog about CEP, this is it! First, thanks to Niels for the endorsement, I don't view the Blogging area as a competition, as people who read Blog tend to read multiple Blogs, and it is also kind of a network of Blogs that sometimes react to each other. To be fair, also most Blogs in this area are marketing oriented Blogs, that they have somewhat different motivation from my Blog, and one cannot compare. Anyway, a good way to start the day.

On two years of the event processing thinking blog

These three pictures are taken from the posting : On unicorn, professor and infant, which is, by far, the most read posting on this Blog. Today is August 28, exactly two years since I started my first posting - this is posting number 314 (PI * 100)

• Event processing technology is still a "child" (maybe no more infant), in the the life-cycle of technologies. It is moving to the youth stage, but have not achieved maturity. It still has challenges in standardization, interoperability with the rest of the universe, engineering and optimization issues, and consumability.
  
• Event processing is not a unicorn, it is not mythological but real, and it has more than one horn, it is not strongly associated with a single application, as some people tended to believe in the past. Each technology has its early adopters, but we see now that the utilization of event processing is pervasive for each industry, no less than technologies like databases, web services and workflows.
  
• The blind people touching an elephant and describing it in a different way is still an excellent metaphor. People have come from multiple perspectives and see the world in a different ways. Our challenge is to get a good understanding of the entire elephant, and overcome this syndrome. This is the mission of the Dagstuhl seminar in 2010.
  

While the unicorn and infant posting was by far the most popular - other popular postings were some postings from 2007 entitled: On event stream processing in which I have written first time that the ESP term has virtually died, and Agnon, the dog, playing and downplaying,
which responded to the claim that I am downplaying the role of rules. Among the 2009 postings the most popular was the one about the event processing language analysis workgroup,


Some more current statistics.

• The number of distinct readers is around 35,000 over these two years, however, many of them are just one-time browsers, the loyal readers, which consist of those who entered the site at least 101 times, their number is around 2,000, if we'll add more than 1,000 more who entered the site between 51-100 times, and another 1,000 who visited between 26-50 times, I would say that there are around 2,000 loyal readers, and around 2,000 more occasional readers - not big numbers in the Internet scale, but fair enough for a Blog focused on a narrow area with a developing community.
  
• The countries with most readers are: USA (still more than 50% of the readers), UK, Germany, Canada, France, India, Australia, Holland, Sweden and Spain. Israel and Japan have many entries, but less readers, which say concentration of loyal readers.
• The biggest referring source, no surprise is Google, other big reference sources are: the TIBCO blog, David Luckham's CEP site, Marco's rulecore Blog, and ebizQ.net.
  

The Blog has been cited by the CEP Knol and has become a major reason for the fact that I was approached by a publisher to write a book.

The other Blogs of the community are either company's Blog - like the TIBCO, Aleri, Streambase and Apama Blogs which express the company's view; or those of independent people or professional bloggers. While I work for IBM, I have never made it as an IBM marketing Blog, but a personal one. I have never been asked by anybody in IBM to write or not to write about anything, and although as IBM employee I do have some restrictions, I have taken upon myself more restrictions as the EPTS chair, among them the restriction of not to write evaluations about products; I am leaving this task to others; the only time that I was asked to revise a posting was fairly recently, when I cited the Gartner Enterprise Architecture Hype Cycle report, where I was asked by Gartner to make clear distinction between what I am citing from Gartner, and what are my own opinions, so I have revised it and restructured the posting in a way the distinction between them will be clear.

Sometimes new topics emerge, and sometime topics are repeating discussions, but never a dull moment in Blogland. Will return with a professional posting s00n.

Concentrating on the important things with event processing

I would like to welcome to the Blogland - Mani Chandy, seen in this picture, one of the pioneers of the event processing area, the person who came to the first event processing symposium in 2006 to tell us what are the obstacles in the way to success in this area, and my partner in organizing the event processing Dagstuhl seminar in 2007, and the second one in 2010.
Mani has recently started a Blog named: Smart systems that sense and respond, so be sure to bookmark it. Mani writes his Blog as mini-papers, including references.

Inspired by the last posting on Mani's Blog I would like to write about a role of event processing that is sometimes overlooked -- not generating more information, but generating less information, or more precisely focus our attention on the right information.




Mani says in this posting that a human attention is the world's scarcest resource, thus focusing the attention on the right stuff has a very strong impact on what's happening. This is true in many senses, I think that in one of the many management courses I took over the years the instructor said something like --- you can do around 20% of what you plan to do, however the difference between success and failure is whether you can identify the right 20%. This is true in many areas of life, a smart student knows what is important to study in depth before the final exam, a smart physician knows which of the symptoms is important, a smart reporter knows how to identify the news item that will take him to fame, and there are many other examples.

Event processing is the basis for smart systems that can detect when it is important to attract the patient attention, in this case the role of the software is not to create more events, but to highlight existing ones, though the highlighted ones may be derived events and not raw events...

So -- focus your attention to other smart articles in the smart systems Blog. More - later.

On Positive Thinking

Two news items have been highlighted in the Israeli press, seemingly unrelated: the first one is Obama's speech in Egypt addressing the middle east people, and the second one is about Dudu Topaz, a famous Israeli entertainer, who was arrested by sending hired people to beat quite hard some senior people in the Israeli media. In the speech that the U.S. president has given earlier this week he addressed the people of this region and called upon them to exercise positive thinking and overcome the differences, to get a permanent settlement. I would say that he called upon the people to desert the "zero sum game" and move to "cooperative game". Some people around me say that this message is very naive, and asserting that since the other side has negative attitude then positive attitude is useless. It is always easier to unite people around negative messages, and indeed the current government of Israel has been elected using negative messages that the people through the collective wisdom of democracy decided to endorse. The other side is not innocent either, generation of children are educated on negative messages of hate. I very much support Obama's call for positive thinking, it will not be easy for people to think that way, after a lifetime of negative thinking, but IMHO this should be the way forward.

The other piece of news was somewhat surprising to most, as written, a famous Israeli entertainer, who was in the past the "king of rating" in commercial TV, hosting a popular show (whom I have never watched). A few years ago, after some embarrassing incidents that he was involved in, the commercial TV stations decided not to hire him anymore, a person like him who felt like a king, with a huge ego, could not stand the humiliation and frustration in repeating rejections and hired some bullies to beat some people hard (they needed hospital treatment and surgeries to undo damages) - one of them was CEO of one of the commercial TV stations, and the other was a program manager of another commercial TV station (the fact that she is a woman did not matter to him), the third was the artist's own agent that did not succeed to take care of him, and the police found in his house plans about several other people. This is an extreme type of negative thinking, but it worthwhile writing something about it. It happens to many people that one day they are stepping out of some position, or some circumstances change, and suddenly they are out of their previous power anymore, some people are going on and doing other things, but some stay on the sideways of the court in which they played before, and feel very frustrated by the fact that they are not the players anymore, since they are convinced that they would have played much better. This lead people to invest a lot of energy in negative thinking and negative actions. Dudu Topaz is an extreme case, but I have seen and am seeing various such cases of investing energy on negative thinking.

I can certainly understand frustration, the wheel is spinning, and as many people, I have been some time up and some time down, but early in my life I decided that positive thinking is much better attitude to life, and looking back is not really a good policy, remembering what happened to Lot's wife when she looked back. Although like any human, I am sometimes tempted to have negative reactions, which mostly proven to be the wrong ones.

I have written in the past about positive thinking in Blog posting, but it is true for other activities as well.

In early stage of my life I have read the poem "IF" by Rudyard Kipling (in Hebrew translation) and felt that this is not just written words, this is a code of behavior that I should adopt. I have returned to cite Kipling from time to time, when I am thinking on a way to behave in extreme situation. I have copied Kipling's poem below his picture.


If (R. Kipling)
If you can keep your head when all about you
Are losing theirs and blaming it on you,
If you can trust yourself when all men doubt you
But make allowance for their doubting too,
If you can wait and not be tired by waiting,
Or being lied about, don't deal in lies,
Or being hated, don't give way to hating,
And yet don't look too good, nor talk too wise:

If you can dream--and not make dreams your master,
If you can think--and not make thoughts your aim;
If you can meet with Triumph and Disaster
And treat those two impostors just the same;
If you can bear to hear the truth you've spoken
Twisted by knaves to make a trap for fools,
Or watch the things you gave your life to, broken,
And stoop and build 'em up with worn-out tools:

If you can make one heap of all your winnings
And risk it all on one turn of pitch-and-toss,
And lose, and start again at your beginnings
And never breath a word about your loss;
If you can force your heart and nerve and sinew
To serve your turn long after they are gone,
And so hold on when there is nothing in you
Except the Will which says to them: "Hold on!"

If you can talk with crowds and keep your virtue,
Or walk with kings--nor lose the common touch,
If neither foes nor loving friends can hurt you;
If all men count with you, but none too much,
If you can fill the unforgiving minute
With sixty seconds' worth of distance run,
Yours is the Earth and everything that's in it,
And--which is more--you'll be a Man, my son!

Back to professional postings -- soon.

Event Processing In Action

Event Processing In Action - this is a title of the book on which I have started to work recently together with my colleague Peter Niblett - although I am typically writing my Blogs always as I and not as We, this time I'll use We in any case that what I am writing refers to Peter as well. In the next few days the first chapter of the book will be made public by the publisher as a green paper. In the picture above you can see a provisional cover of this book, but this is not final yet. The book is planned to be available towards the end of 2009.

The Web 2.0 plays a role in this process, as explained below. Here are some Q&A about it,


What is the motivation for a new book ?

The book has been initiated by Manning Publications, a computing books publisher; their market survey indicated that there is a significant market need for a new book that will articulate and provide a deep dive into the concepts and facilities of event system applications. This book is intended to be the major reference book for enterprise architects, application developers (both technical and semi-technical), and is also expected to be used for instructional purposes (a textbook for a university level course on event processing).

The book written by David Luckham entitled "The power of events" (Addison-Wesley, 2002) has been very influential in setting the initial awareness to the event processing area, and it still is a big inspiration for us; the new book is intended to reflect the contemporary thinking around event processing which has been evolved since 2002.


Why have we agreed to write this book ?

Writing a book is a big responsibility, it is a substantial burden on our time. Furthermore it is a tremendous challenge to produce a high quality book in an emerging area for these target audiences - especially considering the very high expectations that have already been generated around this book. We believe that this book is indeed required, and as technical leaders in the community it is our duty to take this task and help shape the newly emerging discipline of event processing this way. We were also encouraged by our management and colleagues to take this mission.

What is the approach taken in this book ?

The approach taken in the book will not be surprising to the readers of this Blog. Indeed, the book can be considered as a direct descendant of the Blog, it seems that the publisher has approached me based on recommendations of anonymous members of the event processing community that referred him to look at this Blog. I got feedback from others that this Blog is one of the popular sources today to learn what event processing is, but the Blog, as a Blog, is not written in methodical way, it jumps from one topic to another, it treats the various topics in a relatively superficial way, and includes "noise" like this posting; the book should be more focused, getting things in the right order, and in the proper level of depth. The style of writing is similar to that of the Blog.

The book will explain all the event processing concepts by showing step-by-step how a single use case has been constructed. The explanation, like my approach in the Blog, is aimed to be language-style neutral and explain the concepts using a patterns oriented model (although, due to the ambiguity of the term patterns in event processing we use the term building blocks). We are planning to have an appendix in which we will list existing EP products and open source offerings, and provide some high-level details, without providing evaluation or endorsement to any of them. We'll ask for collaboration of the various product owners to get accurate information about their products.


What is the relationship between this book and IBM ?

Both Peter and myself are IBM employees; Peter works in the IBM Hursley Lab in England, where I am working in the IBM Haifa Research Lab in Israel. However we are writing this book (after clearing the legal and managerial permission) as individuals and not as IBM employees; A disclaimer stating that the book represent our opinions and not necessarily the opinion of IBM will be clearly made in the preface to the book, as is done in the top of this Blog. There is a big EP oriented community inside IBM and we hope to get feedback from this community, as part of the feedback from the larger community.


How are Web 2.0 technologies going to impact the authoring process?

As any other book, there is a formal review process, in which the publisher consults with a collection of reviewers representing people from the target audiences, thus most reviewers are architects and developments from various industries, and academic instructors teaching EP courses. In addition, nowadays, book authoring is also considered as an interactive process between the authors and the readers. The MEAP program (Manning Early Access Program) enables readers to interact with the authors through a forum, and contribute comments and questions on the book while being written; when the book will get into the MEAP program I'll further explain it


What are the next steps?


As I have said, Peter and myself are facing with a substantial challenge to create a high-quality book for the readers, and are sure that feedback and reviews from the larger community can help us provide a better book for the target audience; The green paper is due to appear hopefully by the end of this week; I'll post the URL on this Blog as soon as it is available, the MEAP for this book will be set up in the next few weeks. I'll also use this Blog to tell about some dilemmas and challenges in the writing process (another Web 2.0 means of communication).

More -- Later.

On the first step in the way to "event processing manitfesto"

It was a very busy week and alas I had to neglect the blogging hobby, now it is Friday night, I am watching a TV program with old Hebrew songs (my favorite), and decided it is a good time to blog a bit, however, our relatively new cat, who looks somewhat like this (this is not his picture, but of a similar cat I've found on the web) decided that I am a good place to rest on, and did not want to move, another creature who is trying to manage me... He is really a kitten that my daughters found and adopted, and as I have written before, giving names in our family is not an easy task, so he has several names and is known by "the cat". I call him Gilgamesh the terrible.

In 2007 we had the first Dagstuhl seminar on event processing, and we the same set of organizers (Mani Chandy, Rainer von Ammon and myself) decided to apply again for a second Dagsthul seminar in 2010, and the seminar has passed the committee, with some clarifications that we need to provide about scope. I'll let you know if and when it will be finally approved.

The intention of this Dagstuhl seminar (that lasts for 4.5 days) is to have an opportunity for a selected group of people to have a meeting in an isolated place to have in-depth discussions. The proposed goal of this Dagstuhl Seminar is to work on "event processing manifesto". There has been several manifestos of different area in the past, for example: OODB manifesto, Hopefully, by the time of the Dagstuhl seminar we'll have advanced work done by the various EPTS working groups that are being launched this month, and we'll be able to utilize their results in order to better define what "event processing" is -- note that I don't use "complex event processing", and I explained the reasons before.





One of the questions asked is what is the scope of "event processing", since working with events is quite wide area - starting from interrupt handling in operating systems, moving through graphical programming and more -- much of this is related to programming with events in conventional programming, and there are even books dealing with this area. However, our scope is more modest: generic tools for processing events in IT systems. This scope talks on what is needed to build a generic tool, and not ad-hoc programming hard-coded for every single application, and IT systems and not operating system, embedded systems etc..

The illustration above is a first step in thinking about -- what event processing system should include -- parts of it should be mandatory and some optional, however from functionality point of view there are:

• Routing and filtering -- the most basic form of event processing.
• Mediation -- transformation, enrichment, aggregation, split -- the next level of sophistication.
• Pattern Matching --- (I called it in the past "pattern detection") which may involve multiple events from multiple types.

On the bottom of the illustration there are two other entities:

Event processing platforms which are enablers for scalability, distribution and other good qualities. Event processing platforms may have their own functions or host others (or both)...

Pattern discovery that falls under the category "Intelligent Event Processing". It can be done off-line (typically this is the case) or on-line - and then the pattern matching may be unified with the discovery.

In different types of applications we may need different subsets, for example: fraud detection requires pattern discovery, security type detections (e.g. denial-of-service attack or intrusion) may use on-line pattern detection. On the other hand, other applications don't require pattern discovery at all, for example: compliance with regulations, where the regulations are given and cannot be discovered, or BAM systems in which the Key Performance Indicatros are determined according to the corportate strategy and cannot be discovered. Furthermore, there are applications in which pattern matching is not required at all, and all processing is of type filtering, routing, enrichment and aggregation.

And I'll finish with a footnote to David Luckham's recent article. David is trying to answer "critisizm on the Blogsphere" about CEP as a marketing hype, and lack of value from the current set of products. First, I never thought that there is over-hype, on the contrary, relative to the potential of event processing there is under-hype. I am re-posting this illustration taken from Brenda Michelson panel presentation in the last EPTS annual symposium.


The hype is relatively low, and in contrast, the analysts report are all indicating that the EP market has grown by 50% or so in 2008, and IDC even claims that for a second year in a raw that is the fastest growing middleware type. About the Blogsphere crtisizm, as I already written before, much of it stems from diferent interpretations of the term "complex event processing", for example, some of the postings of Tim Bass lead me the conlusion that he believes in the equation : complex event processing = on-line pattern discovery. Again, eliminating the quantification "complex", there is a large set of applications (probably most of the applications I know) of event procssing, do not require stochastic reasoning at all.

I'll post a continuation Blogs about application types, and functions they require.. It is very late - going to sleep.

On external and internal decision in event processing

This picture shows the external and internal parts of an house (not mine...). I am at home now in a day off (have to finish 10 days by the end of the year so taking them one by one), have to travel to Tel-Aviv on some matter, but meanwhile spending a few minutes in Blogland. My last posting has dealt with event processing and decisions.
James Taylor has answered in saying that decisions may be triggered by events, but the decision itself is not dependent on the events. I think that we are on the same wavelength here, so let me make some comments in my terminology. In the early days of active databases there has been a distinction between its use for "internal applications" that helps to manage the DBMS system itself, i.e. manage transactions, and "external applications" which belong to the business domain. Likewise, we can observe internal and external decisions in processing events. The event processing network consists of a lot of internal decisions -- decision of where to route events, decision of which events will be filtered out, decision on when context should expire, decision on whether a pattern has been detected, a decision what to do when a pattern is detected and more. This are all internal decisions that help moving the EPN towards its edges. The interface between the EPN and its consumers occur at that edges. In the edge there are several possibilities:

1. The EPN notifies the consumer about the event that flows in the edge, and the consumer decided manually what should be done, or can manually run any decision process

2. The EPN triggers a decision rule, the decision is a function of the event that triggered it, in this case the EPN triggers an external decision.

3. The EPN triggers a decision, but the decision is not a function of the content of the triggering event, this I assume is the case that James Taylor has referred to.

On event processing as a paradigm shift

The readers are probably familiar with this picture where it shifts between seeing two faces facing each other (in black) and a white vase. I came across a (relatively) new blogger in this area, Pern Walker, blogging for Oracle's "event driven architecture". The title of the posting is:

Event servers, a disruptive technology. It describes the components of the (former) BEA framework, nothing new here, but the interesting part is the conclusion - event processing COTS is a disruptive technology, it displaces custom code in event processing, since it is more cost-effective.

This reminds me of a discussion we had in May 2007 in the Dagstuhl seminar on event processing, it was a night discussion with wine, and was lead by Roy Schulte, the question that Roy has posed to the participants : "Will Event Processing (EDA) become a paradigm shift in the next few years or not?”.

Today, I don't intend to answer this question, instead I'll post part of the discussion in Dagstuhl that included observations about "paradigm shifts" (thanks to my colleague, Peter Niblett, who documented the entire Dagstuhl seminar). I'll return to this topic again, with my (and maybe other) opinions about the answer, after the EPTS event processing symposium

Observations (from the Dagstuhl discussion):

• Paradigm shifts can’t happen if there are too many barriers; have the entry barriers for "event processing" already been removed? ;
• Paradigm shifts are more likely to happen when adopters decide they need a whole new avenue of applications; they are less likely to happen as a way of re-engineering existing systems. For example the German population will reach 1:2 old: young ratio by 2020 so this requires a paradigm shift of healthcare models. Can we identify new avenues of relevant applications?
• Paradigm shifts usually happen as a result of some external change, not just because of innate strengths of the technology itself. Can we identify such external changes?
• Standardization is not necessary for a paradigm shift, but good, appropriate standards (de facto or otherwise) certainly help

Another question is to where in essence is the "paradigm shift" - is it the decoupled "event-driven" paradigm ? is it the "complex event processing", i.e. ability to find patterns on multiple events? is it the entire processing framework as the Oracle's Blog claim?

More - Later

On Unicorn, Professor and Infant

What is common to - unicorn, professor and infant - and how they are related to event processing anyway? -- These where my associations while reading the amusing post of my dear friend Tim Bass when he writes "On elephants and analytics" in an amusing response to my blog posting from this morning .

I am glad that Tim likes my metaphor of the big elephant, since my visit in Thailand last year, I dream about elephants all the time -). Since I am in a metaphors mood today, I'll use some more.

It seems that both Tim and myself share the opinion that the domain of event processing is much larger than the applications that exist today. In fact, I have stated several times, including in my press briefing for the EPTS launch: I believe that the big challenges are still ahead of us, since the industry barely scratched the surface of the potential use

In my previous Blog I have used the metaphor of an infant to describe this state.

Tim is constantly writing about the hype around CEP, again, in the metaphor level, some people that claim that their infant is a professor, and this of course not really true.

Indeed, there are people who over-hype CEP (a perfectly normal phenomenon, there is a hype cycle for each technology, and CEP is getting closer to the top), no surprise here.

Where I respectfully disagree with Tim, is in his claim that what has been done until today is just hype and hence totally worthless, my experience shows otherwise, there are customers which use current products and get value from using these products, and for some strange reason, the number of such customers is even growing (as a comment - calling all of them "time-series stream data engines" is a gross generalization, there is a variety of products, and not all of them are based on time-series streams, on the other hand, those who are based on time-series streams also have value). This does not say that current technologies solve all potential applications, and satisfy all potential users, but treating all the existing technology as a unicorn,

saying that everything is a hype, and there is nothing concrete, like a mythological creature, seems to me as going to extreme. Which takes me back to the elephant that both Tim and myself like While I agree with Tim that those who describe their infant as a professor may be looking at a toy elephant, the other extreme of treating an infant as a unicorn seems also to miss some part of the elephant.

Getting back to issue that start the case analytics and EP -- in my mind - to say that all EP applications require analytic tools, is like saying that one cannot get value from relational databases without using analytic tool, this assertion is certainly true for some types of application, and certainly wrong for others --- and it seems that the same hold for event processing applications. I'll let every reader to quantify the ratio based on his/her experience.

And last comment: it seems to me that there are more than one person in this community who meet customers and care about the value to customer; nobody has a monopoly on understanding all types of customers, applications and business values, and surprise - some vendors actually

believe that bringing value to custotmers is a better way to do business than selling balloons.

On Event Processing Agents

There are different type of agents -- double agents, as seen above which is a series of sweets, insuracne agents, travel agents, and some computerized agents - in my past I have dealt with mobile agents, and there are the intelligent agents in AI, and our own event processing agents (EPA).

David Luckham has written an amusing piece that follows Paul Vincent's "CEP and Agents" in TIBCO's Blog. The amusing part is that Paul has written about AI agents, which uses somewhat different terminology then the event processing terminology, and putting it in a "CEP Blog" is somewhat confusing. I am a "product" of the databases community, and have done some work that was on the AI border in the past, alas, the AI folks are using different terminology to talk about the same thing, and I thought at that time that they are doing it on purpose to confuse me. So, while there are many types of agents, I'll concentrate on the concept of "event processing agent" that has been coined by David Luckham. I like this term and adopted it in the following way: EPA (Event Processing Agents) is a software artifact that receives an event cloud or stream or collection of events or a single event (depends on the agent type and capabilities), does some computation on these event, and produce one or more events as an output. That's it. EPA is also a node in the EPN (Event Processing Network). There are different types of EPAs :

• "simple event processing" EPAs - filter and routing,
• "mediated event processing" EPAs - enrichment, transformation, validation
• "Complex event processing" EPAs - pattern detection
• "intelligent event processing" EPAs - prediction, decisions...

The common denominator: each of them receives events as input, emits events as output and does a single type of function.

I find this type of abstraction both very easy to explain people how EP systems work, and also basis for architecture. The EPN routing can be done by standard middleware, or in a stand-alone mode. Other terminology issues raised by David Luckham is the relationships to the "actor model" and to "engines".

The actor model is a model that helps reasoning about concurrency, while agents in AI are autonomous goal-driven artifacts. These are orthogonal terms, of course. In the context of EPA - when looking at EPAs as an executable network, we can look at each EPA as an actor and apply actor models.

Last but not least -- relationships of EPAs to engines -- an EPA is a software artifcat, it can be an instance of an engine, it can be some software that contains an engine, and it can be hard-coded program, as long as it complies with the EPA definition. In a future world, with inter-operability (and perhaps also language) standards, we'll be able to run (and maybe to self-select) multiple engines for the same EPN, residing in different EPAs.

More about EPA types -- later.

On the spectrum of event processing applications

Back in my office and reading some of the EP Blogs. In the picture above, somebody has tried to draw the spectrum of Blogs (you may want to link to the original in order to see better). One of the last Blogs has dealt again with "simple and complex event processing" claiming that everything done so far in this area is indeed "simple event processing", while real "complex event processing" should support uncertainty and backward chaining. Several posts on this area has been posted by Greg Reemler. We don't have "CEP manifesto" that makes an official definition what is CEP and what is not, and I am not sure that this will be very useful, as it will confuse the customers even more. There is a spectrum of applications that have spectrum of functional and non-functional requirements. On my scientist hat, I am partner to a research work about "uncertainty in event processing" together with Avi Gal and our co-supervised Ph.D. student Segev Wasserkrug However, while there there are applications that require uncertainty reasoning in event processing, there are many others that don't. As I have written several times before, I am not a big fan of the term "complex event processing", due to its ambiguity - some people mean complex processing of events and some mean processing of complex (derived from more than 1) events, some people actually mean complex processing of complex events. I think that we should continue to classify applicatiosn and match the right functional and non-functional requirements to the right applications, but we'll never get to a single functional or non-functional benchmark that will cover all applications in this area. It is better to attract the energy to areas that can help most customers to deal with the problems for which they would like to apply event processing. See my previous posting on : killer applications of EP

More - later.

On Business Intelligence and Event Processing

This is a quiet time, although we do not have holiday period in Israel, those of us, like myself, in which major part of the work requires interaction with the rest of the universe, it is quiet time since my colleagues are away - much less Emails, no conference calls, today I've spent the day in Rehovot known for its famous "Weizmann Institute" but this time (a famous building from the institute is in the picture above) I have visited an IBM site not far this institute, IBM has acquired in the recent years several Israeli companies, and they are now fused into a single lab ILSL - Israel Software Lab (well - ISL was already taken in IBM by India Software Lab).

Anyway, coming back, I saw that the blog world of event processing is also alive these days, but this time I would like to give a few comments to "operational analytics: yesterday today and tomorrow" by Colin White
In this article Mr. White has some interesting assertions (at least, my own interpretation of these assertions):

(1). CEP is a buzzword which stands for a kind of operational analytics that should be used in extreme cases.
(2). BAM has failed since it has been independent of BI
(3). Hint at the fate of CEP (or ESP, it is not clear from his article what is the difference) will be the same if not be part of BI.

While I am sure that Mr. White is a big expert on BI, it seems that he also falls into the "hammer and nail" trap that has been discussed by myself and several other bloggers in this area. So here is some preliminary responses to his assertions:

(1). CEP is a technology, it has roots in multiple disciplines, and some of it has roots in BI, but there is a distance between this and the assertion that it is part of BI. CEP has different uses that may not be even connected to BI (e.g. network management diagnostics or situational awareness), here again we get back to the issue of motivation for using CEP - the consistent view of database people is that the only reason to use CEP is extreme latency/throughput otherwise one can use the right religion of SQL databases, I think that this issue has been discussed also in the EP blogs that there are multiple reasons for using CEP and the high throughput / low latency is one, but not necessarily event the dominant one.

(2). As far as the "BAM has failed" -- is that a fact or a wishful thinking ? in the Gartner BAM and EP summit we have heard some success stories of BAM, and saw some prospering BAM products. While there are synergies and relationships between BAM and BI - I wonder what are the success / failure criteria used to derive this assertion ?

(3). I have already stated my opinion about the question - whether event processing is a footnote to databases
While I spent many years as part of the database community, I am in the opinion that event processing is a discipline by its own right, with some interaction and intersection with the database area, as well as other areas (business rules, distributed computing, middleware, software agents).

This is a preliminary reaction - mostly giving some comments to the article, which does not free me from writing a "positive" article about - the relationships between event processing and business intelligence, and I'll do so in one of the next postings - more later.

EPTS plans for 2008

This picture has been taken in the EPTS meeting in Orlando in September 2007, the meeting itself has been described by Paul Vincent in his blog : first day, second and third day.

As written before the EPTS is a consortium (under construction) intended to promote the understanding and incubate standards in the event processing area. Getting into 2008 - here is the work plan that has been recently approved by the steering committee:

Quarter 1: Launch - as expected when some of the players are big companies, the formalization process take some time, but it will hopefully converge soon. After getting approval of the steering committee - the idea is to invite vendors, customers, academic people and other individuals to join the list of founding members. The launch will also include - introduction of EPTS website (a prototype already exists - thanks to Brian Connell from WestGlobal and Serge Mankovskii from CA) which will support the work of work groups and other EPTS related information (leaving news, blog pointers, general discussion forum for David Luckham's site ). In the launch opportunity we'll also sign and seal the first version of the consensus glossary edited by David Luckham and Roy Schulte, with many contributions provided so far (a draft of this glossary is available)

Quarter 2: "State of the practice" white paper based on a collection of use cases. This is a workgroup launched in the last meeting, with a diversified team of volunteers. The first phase carried out by Tao Lin from SAP, Dieter Gawlick from Oracle and Pedro Bizzaro from University of Coimbra in Portugal to define a template to describe and compare use cases, where the use cases are those presented in the three EPTS meetings, in the Dagstuhl seminar on event processing and maybe open for others. After the glossary, this will be the next community effort in order to understand the state of the practice (there has also been some work last year on "reference architecture" lead by Tim Bass that we'll return to after completing the use cases survey).

Quarter 3: In this quarter we'll hold the fourth EPTS meeting (place and time are not determined yet), participate in the DEBS conference, which we support of becoming the research "flagship" of the event processing community. We'll also launch a portal of teaching material on event processing for the next school year.

Quarter 4: Our first contribution to standard -- we'll launch in early year a workgroup to provide the EPTS input for the OMG RFP on meta-modelling.

Many people are contributing their time and energy for these community activities, and I am looking forward to the momentum that will be created after the EPTS formal launch... stay tuned, or even better - join. Instructions about joining -- soon.

On the envelope for CEP

In two recent blogs - Mark Tsimelzon from Coral8 argued for CEP server vs. embedded CEP libraries, while Paul Vincent from TIBCO argued for the need for infrastructure stack outside the

CEP engine.

Since I avoid making product evaluations in this Blog, I'll talk about the principle.

Conceptually we have a model of "event producer" that produces the events, "event consumer" that consumes the event, and the "event processor" which processes the events and stands in the middle. There are two questions about envelopes:

(1). Is "event processor" an embedded capabilities inside applications or an server.

(2). If "event processor" is a server, is it independent server or part of a larger middleware ?

The answer, as anything is not binary (zero or one), there are cases in which there is a need to have event processing as embedded capabilities (e.g. inside pervasive devices), but Mark is right that the big majority of "event processors" are tend towards the server.

The second question is more interesting --- we have today both stand-alone server and servers that are part of a larger middleware, the rational behind being part of a larger middleware, stems from the fact that event processing is not isolated and has various relations with different applications in the enterprise, it is true that the loose-coupling nature of event driven architectures eases the task of separating it from the applications, but still the integration is the most costly part of building event processing applications, and means to ease the integration has already built into application integration middleware, and if the event processing server is a stand-alone one, there is a need to re-invent this integration, as Paul Vincent rightly say: every $ a CEP vendor spends on middleware integration is a $ less on interesting CEP functionality.

Furthermore, there are some event processing infrastructure and functions (pub/sub, routing - for simple event processing, and ESB mediations like - enrichment, transformation etc, that are already there). Thus, it seems that the ROI will be higher if event processors will be implemented on top of a "middleware stack".

An interesting observation is that from the point of view of application integration middlewares - event processing is becoming a key feature, and there are already some predictions that the standard event processing programming model (which is still not there!), will be the basis for application servers of the future, e.g. Gartner's XTP that I have once discussed and should discuss more.

On - "one size fits all" and Event Processing

Like commercial TV station - if a Blog wants to get "rating" one have to put somewhat controversial - the number of visitors to this Blog has more than doubled in the last few days when I had exchanges of opinions and folk stories with Tim Bass, anyway -- I got tired and did not continue that discussion. One question that I have received somehow related was -- does the fact that I don't think it is worth talking about ESP and CEP as separate entities means that I believe that there is a "one size fits all" in event processing ? well - this is a fair question, in the past I did believe it is true, until I read Mike Stonebraker in his immortal assertion: "One size fits all is a concept whose time has come and gone" Actually, I ceased to believe in it a little earlier, I think that the event processing area is not a monolithic area, and there are some variations needed - however:

• I don't believe that ESP vs. CEP is the right type of partition in this area;
• There may be a need to have various implementation under one roof (the heterogeneous framework approach),

For the first point -- what is the right type of partition ? this is a multi-dimensional questions and we still have to learn more to know the most useful combinations.

One of the important dimensions is the "reason for use" dimension, and here in an internal IBM study we got to five different reasons to use, I'll write about it in one of the next postings.

EPTS has recently launched a workgroup that tries to identify these classifications by doing a comprehansive survey of use cases that will be compared using the same template. A team that consists of Tao Lin (SAP), Dieter Gawlick (Oracle) and Pedro Bizzaro (University of Coimbra, Portugal) is working on this template, and a larger team will handle this survey and analysis -- the end result - a collaborative white paper about the state of the practice in event processing is expected somewhere in the second quarter of 2008. Stay tuned.

More - Later.