The future of Complex Event Processing

Mark Palmer, CEO of StreamBase, has posted a series of predictions about the Complex Event Processing (CEP) market in 2010. Generally, I am not a big fan of making or commenting on predictions related to the technology market but I wanted to add a few interesting thoughts to Mark's list.

For the last few years, I have been actively working with several CEP initiatives involving most of the top CEP technologies in the market. During that time, I have seen CEP evolves all the way from being a fancy experiment to one of the key initiatives in the modern enterprise. I wanted to write a few thoughts about some of the trends that I expect CEP applications in the next couple of years.

CEP starts becoming an enabler for real time business intelligence (BI)

In the last few years, the technologies for implementing real time BI solutions have evolved drastically making it a viable alternative in the enterprise. A clear example is the upcoming Microsoft's PowerPivot stack that is able to process millions of records in a highly optimized multidimensional store that can be accessed from both Excel and SharePoint environments.

The missing component of real time BI solutions are the mechanisms for collecting and processing the data in highly efficient ways. This is where CEP could shine. Most CEP engines are optimized for processing a large number of events using continuous querying mechanisms. In that sense, CEP technologies should become an essential component of real time solutions in order to make these type of solutions a viable alternative in the big enterprise.

CEP moves beyond the financial sector

Undoubtedly, CEP technologies have found a home in the financial sector. Although this tendency is very likely to continue, I do believe CEP is going to start gaining more adoption on other verticals such as manufacturing, the government (military) or the retail industry (RFID). Particularly, the combination of CEP technologies and RFID stacks can enable a large variety of scenarios across various vertical markets.

Big vendors move up the ladder

This is a point on which Mark and I disagree. Personally, I believe that the fact that big technology vendors such as Oracle, Microsoft and TIBCO are embracing CEP is one of the best thing that can happen to the CEP market. Whether they might not have the best technology stacks, it is unquestionable that these vendors will drive innovation and help to make CEP technologies more mainstream.

Another factor to consider is the work vendors such as Oracle, Microsoft and TIBCO are doing integrating CEP stacks into robust middleware and data-storage technologies such as Microsoft BizTalk and SQL Server, Oracle ESB and Oracle DB or TIBCO Event Broker. This level of integration will enable a new set of alternatives for enhancing CEP scenarios.

CEP engines go mobile

Traditionally, CEP applications are based on a server-centric model that is responsible for hosting, managing and scaling the core components of the CEP engines. Even though a large number of CEP scenarios can be implemented using this model, there are native capabilities of CEP engines such as the continuous query engines or adapters that can be very effective on mobile applications. For instance, consider a traditional RIFD scenario on which a mobile reader is capturing thousands of events that need to be filtered based on different patterns. On this scenario, we could read the events using a CEP adapter for the mobile RFID reader and filter them using a CEP continuous query engine.

This post does not intend to be a formalized list predictions about the CEP market. Most of the thoughts listed there are based on tendencies that I have seen in different CEP projects throughout recent years.

What do you think? Am I way off or on the right track? Let's debate :)