Fundamentals

DolphinDB uses stream tables for storing, publishing, and subscribing to streaming data. Stream tables serve as simplified messaging middleware or subscription topics, allowing users to both publish and subscribe to data streams. Inserting a record into a stream table is equivalent to publishing a message from the data source.

Stream tables are essentially specialized in-memory tables. They support table query with SQL statements, the key distinction is that records in stream tables cannot be modified or deleted.

DolphinDB uses a publish-subscribe (Pub/Sub) communication model to enable publishing and subscription of streaming data through message queues. Pub/Sub lets you create systems of event producers and consumers, called publishers and subscribers. This model decouples stream publishers and subscribers, allowing independent development and management of each module and enhancing system scalability while improving response efficiency for publishers.

Streaming engines are built-in computational functions for specific stream processing scenarios, such as real-time factor calculation, and stream joining. Simple tasks can be handled by a single engine, and complex processing tasks can be implemented by creating streaming pipeline.

DolphinDB's modular and extensible stream processing framework uses core components like stream tables and engines as reusable modules. These can be flexibly combined in a publish-subscribe manner as building blocks to form powerful processing pipelines.

Data ingested into DolphinDB for stream processing can be divided into the following ways:

• Data Writes: Real-time streams written via DolphinDB, plugins or API methods.
• Messaging Plugins: Data streams from third-party sources through messaging plugins.
• Data Replay: Historical batch data from DolphinDB databases and tables.

DolphinDB enables simultaneous subscriptions to streaming data by multiple consumers. Subscriptions can be set up through the following methods:

• Server subscribeTable function:
  • DolphinDB tables for data warehousing.
  • Streaming engines for real-time analysis.
  • Third-party consumers, such as analytic systems and visualization platforms.
• API subscription: API clients (e.g., Python applications) can subscribe to data streams, enabling integration into various application use cases.