Complex Event Processing

What is DolphinDB CEP

Starting from version 3.00.0, DolphinDB incorporated Complex Event Processing capabilities through the introduction of the DolphinDB CEP. It allows you to capture and match streams of events against predefined patterns, providing insights into ongoing occurrences.

The CEP engine enables you to:

  • Specify the patterns that you want to detect and act upon in your stream;

  • Monitor streams of events to find particular events or patterns of interest;

  • Perform actions such as aggregation and transformation based on particular events or patterns;

  • Extract information from event streams and find out relationships between them.

Differences between CEP engine and other streaming engines:

CEP

Other Streaming Engines
Data Sources Receives events of various types. Each event can be defined with different schema. Receives streams with same schema.
Data Processing Event-driven, all processing is triggered in response to events. The same set of calculation rules applied on all incoming data.
Application Designed to continuously analyze real-time streams, extracting and identifying specific events or relationships between them. Designed to process streams with aggregation, factor calculation, etc.

CEP Workflow

The DolphinDB CEP is composed of the following main components:

  • Stream Event Serializer: Serializes real-time event streams from various data sources and writes them to a heterogeneous stream table in DolphinDB.

  • Stream Event Deserializer: Deserializes the events from the subscribed stream tables. It is implemented by the CEP engine internally.

  • Event Dispatcher: Distributes events to sub-engines based on specific attributes (defined within the CEP engine).

  • CEP Engine: Receives the latest event data in real-time by subscribing to the heterogeneous stream table. The CEP engine will implement event matching logic and act on selected events. The CEP engine can generate multiple sub-engines for parallel processing of events. Each sub-engine consists of:

    • Event Input Queue: Holds the incoming events for each sub-engine.

    • Event Matcher: Analyzes the events from the input queues to identify predefined patterns or conditions of interest.

    • Monitors and Listeners: Monitors define the processing logic for events and listeners listen for the events passed to the event matchers with specific patterns or conditions. When the event matchers identify events that match the patterns defined in the event listeners, the corresponding actions will be invoked.

    • Event Output Queue: Holds the output events from each sub-engine. The Stream Event Serializer serializes these output events and writes them to a heterogeneous stream table, which can then be subscribed to by APIs (clients) or plugins.

The following figure illustrates the workflow of DolphinDB CEP.