Some events caused by manual mistakes or glitches, disrupt an organization’s process flow, leading to significant losses. Identifying root causes after these catastrophic results occur does not prevent future events since processes can be impacted in numerous ways. Organizations leverage complex event streaming techniques and tools to identify the current process status by examining streaming data.
What is complex event processing (CEP)?
Complex event processing is a real-time alerting hub of the organization. Complex event processing is connected to the various sources of data in the business with a series of pre-built triggers for certain events. When certain events happen, specified actions (e.g. alerts) can be triggered.
Complex event processing is a subset of event stream processing (ESP). Here are the Wikipedia definitions of both:
Complex event processing, or CEP, consists of a set of concepts and techniques developed in the early 1990s for processing real-time events and extracting information from event streams as they arrive. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) in real-time situations and respond to them as quickly as possible.
Event stream processing, or ESP, is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing (CEP).
How does it work?
Complex event processing is connected to the various sources of data in the business with a series of pre-built triggers for certain events. When certain events happen, specified actions (e.g. alerts) can be triggered. Its working flow can be simplified as: “If events A, B, C and then take action B”.
What are CEP use cases?
Complex event processing is a good fit for cases where there is a high volume of data from different sources and organizations need to take real-time action on this data. CEP approaches are embedded in different products and implemented to support machine learning solutions with rules-based logic. Therefore, they are rarely identified as separate products in these use cases. Here are the most common use cases we’ve encountered during our research:
- Buy/Sell decision in stock market trading: Complex event processing applications can process incoming stock price data. During the data processing procedure, the application matches stock data with historical and real-time patterns and automates the sell or buy decision process.
- Predictive maintenance: With the help of incoming data from IoT sensors, manufacturing facilities can decide whether there are any patterns indicating it is necessary to shut down or perform maintenance work on equipment.
- Real-time marketing: Complex event processing can also be leveraged as a marketing tool to better understand customer behavior within a product or website. CEP applications can recognize meaningful patterns from the trails that customers leave while spending time on the website. For example, when CEP identifies a pattern about a customer, it can deliver specific promotions or recommendations.
- Autonomous vehicles: Self-driving cars collect real-time data with sensors. For example, with the help of complex event processing, vehicles can recognize dangerous situations and pass control to humans.
- RFID data in Retail: Retailers can place RFID tags on items to collect and process real-time data for stock tracking, transaction, inventory level or individual purchase order history. Instantaneous and continuous monitoring of items provide insights about store management so that retailers can optimize their store layout efficiency.
In general, complex event processing tools can focus on any business process data from any business line (IT, accounting, recruitment, call center, etc.) to identify problems in the organizations’ processes.
What are the case studies?
In IBM‘s case study, a bank offers customizable alerts to their customers for their activities such as account activities, stock market activities, currency exchange rates. Customers can choose the type of alerting mechanism (e.g. mail, SMS) and the frequency in which the alerts will be triggered (e.g. during business hours immediately, otherwise in the next business day). Here are sample alerts that customers may use:
- Notify when there has been a withdrawal of a total of $X from an account within Z hours. (X and Z are definable parameters at subscription to the alert.
- Notify when there were at least X ATM withdrawals of more than $Y within Z hours. (X, Y and Z are definable parameters at subscription to the alert)
What are the leading tools for complex event processing?
- Amazon Kinesis Analytics
- Apache Spark Streaming used by Databricks
- Apache Flink used by data Artisans
- Apache Samza used by LinkedIn
- Apache Storm used by Twitter
- EVAM Event Streaming Platform
- Fujitsu Software Interstage Big Data Complex Event Processing Server
- IBM Streams, Operational Decision Manager (ODM)
- Microsoft Azure Stream Analytics, Stream Insight
- Oracle Stream Analytics and Stream Explore
- Red Hat Decision Manager
- TIBCO Streaming
If you still have questions about complex event processing, don’t hesitate to contact us.