DolphinDB Order Book Engine: Build High-Frequency Order Books from Tick-by-Tick Data

The order book engine employs an event-driven state update mechanism to build the limit order book from tick-by-tick order and trade data.

• Within each channel, events are processed sequentially based on ApplSeqNum, the exchange-assigned message sequence identifier (also known as BizIndex), which is independently enumerated starting from 1 per channel.

• The order book state is maintained through incremental updates: Upon receiving an order event, the corresponding price and volume are recorded in an internal state cache. Upon receiving trade or cancellation events, the matched quantities are decremented from the cached order state.

• Through continuous event-driven updates, the engine maintains a real-time view of order quantities, order counts, and aggregated depth information at each price level.

This building logic is further adjusted according to exchange-specific trading rules before generating the final order book output, ensuring consistency with market regulations. For example, for SZSE stocks, the engine implements ChiNext price band constraints. Orders violating price limits may be excluded from the order book, and this behaviour can be configured via runtime parameters.

The correctness of the building process can be validated by comparing the generated order book with the exchange's 3-second snapshots. The validation methodology and implementation details are presented in Chapter 5.

An order book snapshot represents a cross-section of the order book at a specific point in time. The definition of the time window and the triggering of snapshot generation are fully managed by the engine. This section describes the underlying mechanism for time windowing and snapshot triggering.

When the order book engine is configured to generate snapshots at a 1-second interval, a snapshot with timestamp 10:00:00 represents the latest order book state constructed from all tick-by-tick data with timestamps less than or equal to 10:00:00. Window-based metrics, such as interval trade volume, are calculated over the interval (09:59:59, 10:00:00]. The snapshot at 14:57:00 follows slightly different semantics. Since the engine stops processing before the closing call auction session, the final snapshot excludes ticks at 14:57:00 itself and corresponds to the interval (14:56:59, 14:57:00).

In real-time processing, tick-by-tick data is ingested sequentially, and the engine must determine when a window is complete. Taking the 10:00:00 snapshot as an example, the triggering rule is as follows: once a tick with a timestamp greater than 10:00:00 is received (either order or trade), the engine finalizes and outputs the 10:00:00 snapshot. The triggering tick itself is excluded from the computation. This rule can be applied either per symbol or across all symbols. When applied per symbol, each symbol triggers computation and output of its own order book only after the engine receives a tick-by-tick record for that symbol with a timestamp greater than 10:00:00. When applied across all symbols, a tick-by-tick record from any symbol with a timestamp greater than 10:00:00 triggers computation and output of the order books for all symbols.

Additionally, if tick-by-tick data is missing for a given period, no output will be generated for that interval. For example, during the midday trading break, no snapshots are produced for the interval between 11:30:01 and 12:59:59 due to the absence of market data.

By strictly aligning snapshot generation with tick timestamps, the system ensures consistency between real-time and historical computations, achieving unified stream and batch processing. It is recommended to treat all symbols within a single channel as a whole and use event time (i.e., the trade timestamp in the data) to determine snapshot boundaries. This approach ensures both timeliness and accuracy of snapshot generation. In future releases, the engine will also support processing based on machine time, where the latest order book is output at fixed system time intervals.

In live trading, tick-by-tick data ingested into the system may arrive out of order. Improper handling of such a disorder can compromise the correctness of order book building. Within exchange systems, each order and trade event in a given channel is assigned a sequential identifier, ApplSeqNum, which reflects the true execution order of events on the exchange side. Out-of-order data refers to cases where events do not arrive at the processing system in strict ApplSeqNum order, typically due to network latency or other transmission issues. As a result, incoming tick-by-tick data may need to be reordered to restore the original event sequence.

For example, if a trade event for Symbol A arrives before its corresponding order event, and the engine processes events strictly in arrival order, the resulting order book state will be incorrect. The order book engine supports ApplSeqNum-based ordering. The engine detects sequence gaps and temporarily buffers out-of-order data until the missing sequence entries arrive, after which computation and output are performed. You can choose whether to enable out-of-order data handling based on your business requirements and the quality of the data you receive.

The table below illustrates how events with different ApplSeqNum values are processed, where the sequence from left to right represents increasing event order, i.e., ApplSeqNum = 1 is processed first.

Since version 2.00.12, the order book engine supports stocks, funds, and convertible bonds across both SSE and SZSE. The detailed rule coverage is as follows:

• Support for ChiNext price band constraints can be enabled via configuration. For SZSE stocks, the default behaviour follows pre–full registration system rules, including ChiNext price band logic.

• The engine implements handling for special order types, including market orders and best-bid/ask orders, ensuring they are correctly reflected in the built order book.

• Convertible bonds in both SSE and SZSE are processed according to the new regulations introduced in August 2022.

• Special rules applicable to ChiNext stocks during the first five trading days after listing are not included; therefore, order book correctness is not guaranteed for ChiNext instruments in this period.

• Special rules for main board stocks on their listing day are not included; therefore, order book correctness is not guaranteed for main board instruments on the first trading day.

• Merged trade and order input table

The engine accepts a single input table that contains both tick-by-tick trade and order data. Trade and order records must be normalized into a unified schema so they can be stored in the same table.

• Required input fields

The input data must contain at least 10 mandatory fields. These fields are essential for engine computation. During engine initialization, the parameter inputColMap must be specified. It is a dictionary that maps required logical fields to actual column names in the input table. Additional fields may be included in the table, but are ignored by the engine.

• Enum value constraints

  The columns listed above are defined using predefined enum values. Input values must strictly follow the defined conventions to ensure correct results. For example, in the BSFlag field, 1 represents a buy order and 2 represents a sell order.

• Notes on the seqColumn Column

  • For SZSE order data, the seqColumn represents the order identifier. It is required for correctly matching subsequent cancel and trade events to the original order, and therefore must be accurate.

  • Moreover, seqColumn can also be used to determine processing order: If you create the order book engine with orderBySeq=false, the engine processes input records one by one in the order they arrive. If you create the engine with orderBySeq=true, the engine processes data one record at a time in ascending order of the values in the seqColumn column. For details, see Section 1.3.

  • SZSE stocks, SZSE convertible bonds, SZSE funds, and SSE convertible bonds each use separate channels. ​SSE funds and stocks share the same channel. ​Within a channel, ApplSeqNum is numbered continuously starting from 1.

• A single order book engine instance can process at most one full channel of instruments per trading day.

The engine maintains a cumulative order book state for each instrument and does not reset the state at day boundaries. Therefore, cross-day processing is not supported. When triggerType=”mutual”, processing multiple channels in a single engine is not recommended, as merging multiple channels by ApplSeqNum may break event ordering and lead to early snapshot triggers.

Tick-by-tick trade and order data are typically stored in separate files or database tables with different schemas. This section provides examples of how to merge trade and order data into an input table for batch processing. Due to differences between SZSE and SSE data formats, the processing is described separately in the following subsections.

When ingesting real-time data into the order book engine, it is recommended to write tick-by-tick trade and order data into the DolphinDB stream tables in the exact order in which the exchange sends them. If the arrival order at the DolphinDB server is out of order with respect to ApplSeqNum, the engine can be configured with orderBySeq=true to enforce ordered processing based on the seqColumn field. The typical real-time ingestion pipeline is as follows:

• First, create a stream table in DolphinDB, either as a shared stream table or a persisted stream table.

• Next, subscribe to both tick-by-tick trade and order data via a market data SDK. In the SDK callback function, parse incoming messages and convert enum values as required.

• Finally, write the tick-by-tick trades and tick-by-tick orders from the same channel into the same stream table.

There are two common approaches for integrating real-time market data into the DolphinDB server:

• Approach 1: Market data plugins such as amdQuote and INSIGHT already support writing trade and order data into a single table and include built-in enum conversion. With the amdQuote or INSIGHT plugin, you can receive input data directly in the format expected by the order book engine.

• Approach 2: DolphinDB API integration. Alternatively, external applications can be developed using the DolphinDB API. It is recommended to use the MultithreadedTableWriter interface for high-throughput real-time ingestion. Implementation details can follow the pipeline described above and reference the source code of existing plugins.

First, create an order book engine. In the following example, the engine named “demo” generates a 10-level order book for SZSE stocks at a 1-second frequency, with additional tick-level trade detail fields.

// Get the schema of the order book engine output table
depth = 10
orderBookAsArray =true
outputColMap, outputTableSch = genOutputColumnsForOBSnapshotEngine(basic=true, time=true, depth=(depth, orderBookAsArray), tradeDetail=true, orderDetail=false, withdrawDetail=false, orderBookDetailDepth=0, prevDetail=false)
// Define the engine parameter outputTable, which specifies the output table
outTable = table(1:0, outputTableSch.schema().colDefs.name, outputTableSch.schema().colDefs.typeString)
// Define the engine parameter dummyTable, which specifies the schema of the input table
colNames = `SecurityID`Date`Time`SourceType`Type`Price`Qty`BSFlag`BuyNo`SellNo`ApplSeqNum`ChannelNo`ReceiveTime
colTypes = [SYMBOL, DATE, TIME, INT, INT, LONG, LONG, INT, LONG, LONG, LONG, INT, NANOTIMESTAMP]
dummyOrderTrans = table(1:0, colNames, colTypes)
// Define the engine parameter inputColMap, which specifies the meaning of each field in the input table
inputColMap = dict(`codeColumn`timeColumn`typeColumn`priceColumn`qtyColumn`buyOrderColumn`sellOrderColumn`sideColumn`msgTypeColumn`seqColumn`receiveTime, `SecurityID`Time`Type`Price`Qty`BuyNo`SellNo`BSFlag`SourceType`ApplSeqNum`ReceiveTime)
// Define the engine parameter prevClose, which specifies the previous closing price. prevClose does not affect any field in the final output except the previous closing price itself
prevClose = dict(`000400.SZ`300274.SZ`300288.SZ`300122.SZ`300918.SZ, [1.1, 2.2, 3.3, 4.4, 5.5])
// Create the engine to calculate and output a 10-level order book for SZSE stocks every second
engine = createOrderBookSnapshotEngine(name="demo", exchange="XSHE", orderbookDepth=depth, intervalInMilli=1000, date=2022.01.10, startTime=09:30:00.000,  prevClose=prevClose, dummyTable=dummyOrderTrans, inputColMap=inputColMap, outputTable=outTable, orderBySeq=false, outputColMap=outputColMap, orderBookAsArray=orderBookAsArray)

• Unlike the previous subsection, this example specifies the outputColMap parameter when calling createOrderbookSnapshotEngine, which is used to select the required output fields. In Section 3.1, outputColMap was not specified, so the engine returned the default output schema. For details on the usage of EoutputColMapE and the default output format, t, please refer to CcreateOrderbookSnapshotEngine.

• To facilitate field selection, DolphinDB provides the function genOutputColumnsForOBSnapshotEngine, which returns both the required output fields and the corresponding output table schema. The function accepts a configuration that specifies which categories of fields to include. In this section, basic, time, depth, and tradeDetail are set to true, and all others are set to false, which means only the basic trade information, time fields, order book fields, and trade detail fields are required.

• When time=true, the genOutputColumnsForOBSnapshotEngine function returns multiple time fields, which are also supported by the engine as optional outputs. These fields are primarily used for performance analysis, including system timestamps at the moment of result generation and the original tick arrival time that triggered the computation. To support this, the inputColMap must include the key ReceiveTime, enabling the engine to capture the arrival time of each tick-by-tick record.

• In addition, orderBookAsArray is set to true, which means the multi-level price and volume data are output as array vectors. For example, the 10 bid prices are stored in a single field. Otherwise, each level would be stored in separate fields.

Next, load the historical data into the order book engine. Compared with the previous section, the input data in this example includes an additional column, ReceiveTime, which represents the arrival time of each tick-by-tick record.

t = select * from loadText("./orderTrans.csv") order by ApplSeqNum
update t set ReceiveTime = now(true) // Construct the receive time column
getStreamEngine("demo").append!(t)

The output table outTable is as follows:

First, create an order book engine. In the following example, the order book engine named "demo" generates a 10-level order book for SZSE stocks at a 1-second frequency, with four additional user-defined metrics.

// Define the order book depth and other parameters
depth = 10
orderBookAsArray =true
outputColMap = genOutputColumnsForOBSnapshotEngine(basic=true, time=false, depth=(depth, orderBookAsArray), tradeDetail=true, orderDetail=false, withdrawDetail=true, orderBookDetailDepth=0, prevDetail=false)[0]
// Define the engine parameter dummyTable, which specifies the schema of the input table
colNames = `SecurityID`Date`Time`SourceType`Type`Price`Qty`BSFlag`BuyNo`SellNo`ApplSeqNum`ChannelNo
colTypes = [SYMBOL, DATE, TIME, INT, INT, LONG, LONG, INT, LONG, LONG, LONG, INT]
dummyOrderTrans = table(1:0, colNames, colTypes)
// Define the engine parameter inputColMap, which specifies the meaning of each field in the input table
inputColMap = dict(`codeColumn`timeColumn`typeColumn`priceColumn`qtyColumn`buyOrderColumn`sellOrderColumn`sideColumn`msgTypeColumn`seqColumn, `SecurityID`Time`Type`Price`Qty`BuyNo`SellNo`BSFlag`SourceType`ApplSeqNum)
// Define the engine parameter prevClose, which is the previous day's closing price. prevClose does not affect any fields in the final output other than the previous day's closing price itself
prevClose = dict(STRING, DOUBLE)
//// Define user-defined factors
def userDefinedFunc(t){
        AvgBuyDuration = rowAvg(t.TradeMDTimeList-t.TradeOrderBuyNoTimeList).int()
        AvgSellDuration = rowAvg(t.TradeMDTimeList-t.TradeOrderSellNoTimeList).int()        
        BuyWithdrawQty = rowSum(t.WithdrawBuyQtyList)
        SellWithdrawQty = rowSum(t.WithdrawSellQtyList)
        return (AvgBuyDuration, AvgSellDuration, BuyWithdrawQty, SellWithdrawQty)
}
// Define the output table for the order book engine
outputTableSch = genOutputColumnsForOBSnapshotEngine(basic=true, time=false, depth=(depth, orderBookAsArray), tradeDetail=false, orderDetail=false, withdrawDetail=false, orderBookDetailDepth=0, prevDetail=false)[1]
colNames = outputTableSch.schema().colDefs.name join (`AvgBuyDuration`AvgSellDuration`BuyWithdrawQty`SellWithdrawQty)
colTypes = outputTableSch.schema().colDefs.typeString join (`INT`INT`INT`INT) 
outTable = table(1:0, colNames, colTypes)
// Create the engine to generate a 10-level order book output for SZSE stocks every second
try{dropStreamEngine(`demo)} catch(ex){}
engine = createOrderBookSnapshotEngine(name="demo", exchange="XSHE", orderbookDepth=depth, intervalInMilli=1000, date=2022.01.10, startTime=09:30:00.000,  prevClose=prevClose, dummyTable=dummyOrderTrans, inputColMap=inputColMap, outputTable=outTable, outputColMap=outputColMap, orderBookAsArray=orderBookAsArray, userDefinedMetrics=userDefinedFunc)

• When creating the engine, the userDefinedMetricspara meter is specified. It is a unary function that defines the user-defined metrics. The input of this function must be a table, where each row represents a single instrument snapshot, and each column corresponds to an internal engine metric defined in outputColMap. You can use these built-in metrics to derive user-defined metrics. In this example, trade and cancellation details between two consecutive order book snapshots are used to derive metrics such as order duration and cancellation volume within the interval.

• Note: Once userDefinedMetrics is specified, the output schema is no longer fully aligned withoutputColMap. Instead, the output consists of two parts:

  • Built-in metrics returned by the genOutputColumnsForOBSnapshotEngine function, including basic and depth fields.

  • User-defined metrics computed via userDefinedMetrics.

Finally, historical data is loaded into the order book engine.

t = select * from loadText("./orderTrans.csv") order by ApplSeqNum
getStreamEngine("demo").append!(t)

The output result table outTable is shown below, where the highlighted section represents user-defined metrics:

We directly ingested one full trading-day channel of SZSE data covering over 600 stocks into the order book engine and measured execution time using the timer statement. The results show that generating a 1-second order book requires approximately 1 minute and 57 seconds. Since the data volume of 10-millisecond snapshots is more than three times that of 1-second snapshots, the increased output overhead leads to higher total processing time for 10-millisecond resolution.

Test results:

Test environment:

DolphinDB: 2.00.12 2024.04.02

Hardware:

• Kernel: Linux 3.10.0-1160.el7.x86_64 (x86_64)

• CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

• Memory: 503 GB

We tested the computation latency in live trading using the insight plugin to ingest full-market SZSE tick-by-tick data. Data was published via a pub-sub mechanism and written into the order book engine in real time. Because SZSE stocks are distributed across four channels, we created four separate order book engines and deployed them across four consumer threads. The result shows that the average latency of a 1-second order book snapshot is approximately 0.67 milliseconds, meaning the latest snapshot can be generated within 0.67 ms after receiving the tick-by-tick data.

Test results:

Note: The reported latency includes both the engine's computation time and data ingestion latency from publishing to the engine. In live trading, response latency can be calculated as: UpdateTime2 - UpdateTime, where UpdateTime2 is the time when the engine finishes the computation, and UpdateTime1 is the time when the system receives the tick-by-tick data (the ReceiveTime field). For each snapshot, UpdateTime1 is set to the receive time of the tick-by-tick record that triggers the snapshot output.

Test environment:

DolphinDB: 2.00.12 2024.04.02

Hardware:

• OS: CentOS Linux release 7.9.2009 (Core)

• Kernel: Linux 3.10.0-1160.el7.x86_64 (x86_64)

• CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

• Memory: 503 GB

Check that the values in the abnormal field of the output table are all false. This indicates that no unmatched order references occurred during building, i.e., no trades or cancellations were received without corresponding order records. A successful check implies that the input data contains neither missing records nor out-of-order events. The following script validates the results, where outTable is the output table generated by the order book engine.

outTable = loadText("./engineOutput.csv")
@testing: case = "abnormal"
assert (exec count(*) from outTable where bool(abnormal)=true)==0

After execution, no output indicates that the validation passed. If an abnormal is returned, the validation failed.

After passing the checks in Section 5.1, the results can be validated against the exchange-provided 3-second snapshots.

Direct timestamp joins should not be used for comparison, because exchange snapshots and tick-by-tick data are published asynchronously. As a result, their timestamps do not match. An order book snapshot represents the state of the order book at a particular instant. Given sufficiently dense generated snapshots, the exchange of 3-second snapshots should appear somewhere within the generated high-frequency snapshots. Therefore, correctness is evaluated based on whether the generated snapshots fully cover the exchange snapshots.

Using SZSE stocks as an example, the engine generates snapshots at 10 ms intervals. The accuracy is evaluated based on the coverage rate of exchange 3-second snapshots by the generated 10 ms snapshots. A 3-second snapshot is considered covered if a generated 10 ms snapshot can be found with identical bid and ask prices and sizes across all 10 levels, and the timestamps of all matched 10 ms snapshots are strictly increasing.

Input data

The following example uses SZSE data only:

Engine-generated snapshot:engineOutput10s.zip

Exchange 3-second snapshot:tick.zip

Note that the data files are compressed archives. Please extract them before use.

Comparison script

The following scripts validate the results for a single stock over one trading day.

First, join the two order book datasets:

// Log in
login("admin", "123456")
// Specify the stock code and trading date to be validated
id ="300274.SZ"
compareDate=2022.06.01
// Load the exchange-provided 3-second snapshot data file
orgTickPath = "./tick.csv"
originTick = select * from loadText(orgTickPath) where SecurityID=id and MDDate=compareDate and ((MDTime between 09:30:00.000:11:30:00.000) or (MDTime between 13:00:00.000:14:56:59.000)) order by MDTime
// Load the 10 ms snapshot data generated by the order book engine
genTickPath = "./engineOutput10s.csv"
generateTick = select * from loadText(genTickPath)  where SecurityID=id and date(timestamp)=compareDate order by timestamp
// Define the field names to be compared in the exchange snapshot data
orgColNames = `SecurityID`MDTime`LastPx`HighPx`LowPx`TotalBidQty`TotalOfferQty join ("BuyPrice" + string(1..10))  join ("BuyOrderQty" + string(1..10)) join ("BuyNumOrders" + string(1..10)) join ("SellPrice" + string(1..10)) join ("SellOrderQty" + string(1..10)) join ("SellNumOrders" + string(1..10)) 
// Define the field names corresponding to the snapshots generated by the order book engine.
genColNames = `SecurityID`timestamp`lastPx`highPx`lowPx`bidVol`askVol join ("bids" + string(1..10))  join ("bidVolumes" + string(1..10)) join ("bidOrderNums" + string(1..10)) join ("asks" + string(1..10)) join ("askVolumes" + string(1..10)) join ("askOrderNums" + string(1..10)) 
// Define a function to concatenate multiple numeric fields in a snapshot into a single string for comparison
def calContent(contentColNames, tickTable, format="0.00"){
	contentStr=concat("format(" +contentColNames[2:] +", \""+format+"\")", "+\",\"+")
	contentSql=parseExpr(contentStr)
           	return sql(select=(sqlCol(contentColNames[0]),sqlCol(contentColNames[1]),sqlColAlias(contentSql, "Content")), from=tickTable).eval()
}
// Generate the Content string for the exchange 3-second snapshot data
originContent=calContent(orgColNames, originTick, "0.00")
// Generate the Content string for the engine-generated 10 ms snapshot data
generateContent=calContent(genColNames, generateTick, "0.00")
// Perform a left semi-join to compare the two snapshot tables, using the exchange 3-second snapshots as the left table
jointable =  select * from lsj(originContent as a ,generateContent as b, `Content, `Content)

• originTick is the exchange-provided 3-second snapshots, and generateTick is the 10 ms snapshots generated by the order book engine.

• orgColNames and genColNames are vectors. The first two elements are the stock ID and date columns, and the remaining elements specify the metric fields used in comparison. Metrics associated with the current aggregation window, such as interval trade volume, should not be included because snapshots at different frequencies cannot match on such fields.

• The calContent function concatenates multiple metric fields into a single string field named content. The format parameter specifies data precision. For stocks, set it to ”0.00” for two decimal places. For convertible bonds, set it to “0.000” for three decimal places.

• Finally, use lsj to perform a left semi join on the content fields of the two snapshot tables, with the 3-second snapshot table as the left table. The resulting jointable has the same row count as the 3-second snapshot table.

The following script counts and validates the joined result table:

• Test case 1 checks whether the coverage rate of 3-second snapshots by generated 10 ms snapshots is at least 99%. If it fails, it returns “Not Covered”.

• Test case 2 checks whether all generated 10 ms snapshots corresponding to each 3-second snapshot are ordered by timestamp. If it fails, it returns “Not In Order”.

// Assert that the coverage rate of 3-second snapshots by generated 10 ms snapshots is greater than or equal to 100, indicating the test passes
@testing: case = "Not Covered"
assert (exec count(b_SecurityID)\count(*)*100 as coverage from jointable ) >= 99
// Assert that all generated 10 ms snapshots for each 3-second snapshot are ordered by timestamp, indicating the test passes
flag = true 
if(at(deltas(jointable.timestamp)[1:]<=0).size()==0) {
	break
}else { 
	leftTime = 09:15:00.000
	for (row in originContent){
		genCon = exec Content from generateContent where time(timestamp)>= leftTime 
		genTime = exec time(timestamp) from generateContent where time(timestamp)>= leftTime 
		findGenTime = genTime[genCon.find(row.Content)]
		if (findGenTime==NULL){
			flag=false
			break
		} else { 
			leftTime=findGenTime
		}
	}
}
@testing: case = "Not In Order"
assert flag

If validation fails, the number of compared fields can be reduced to identify problematic metrics.

Common causes of validation failure include:

• Incorrect preprocessing of input data. Refer to Chapter 2 for the correct preprocessing procedure.

• Precision differences between engine output and exchange snapshots. For example, the engine may output total turnover as a floating-point value, while some data vendors provide integer values.

• Differences between null values and 0. In the sample script above, comparisons between null and 0 are treated as mismatches. The following script can be used to replace all null values with 0 before validation:

originTick = nullFill(originTick, 0)
generateTick = nullFill(generateTick, 0)

The optional engine parameter outputIntervalOffsetMap is a dictionary that specifies per-instrument time offsets for triggering snapshot outputs. The dictionary key is a string indicating the instrument ID, and the dictionary value is an integer indicating the offset in milliseconds. Using the 1-second order book as an example, assume three instruments: id1, id2, and id3. You can specify the outputIntervalOffsetMap parameter (dict(["id1","id2","id3"], [0, 50, 50])) to achieve the effect shown below.

In this configuration, snapshots for different instruments are generated at different times. id1 produces output at hh:MM:ss.000, while id2 and id3 produce outputs at hh:MM:ss.050, i.e., 50 ms after each whole-second boundary.

Proper use of this parameter provides several benefits. First, reducing the number of simultaneous outputs decreases the processing time per batch of snapshots. More importantly, it smooths the output distribution across time, reducing pressure on downstream consumers and improving resource utilization. Without this mechanism, large volumes of data may be output at the same timestamp, causing bursts of heavy load followed by idle periods.

The following modifications are made to the code in Section 3.1:

outputIntervalOffsetMap =dict(`300288.SZ`300122.SZ`300918.SZ, [50, 50, 50])
engine = createOrderBookSnapshotEngine(name="demo", exchange="XSHE", orderbookDepth=10, intervalInMilli=1000, date=2022.06.01, startTime=09:15:00.000, prevClose=prevClose, dummyTable=dummyOrderTrans, inputColMap=inputColMap, outputTable=outTable, outputIntervalOffsetMap=outputIntervalOffsetMap)

The stocks 300288.SZ,300122.SZ, and 300918.SZare configured to output snapshots 50 ms after each whole-second boundary, while all other stocks, for which no offset is specified, continue to output at whole-second boundaries.

No. The required fields can be identified via the inputColMap parameter, which defines the mapping between logical fields and input columns.

No. The output fields are specified through the outputColMap parameter. However, the schema of outputColMap must be consistent with the defined output schema.

When the engine processes trade or cancellation events, it attempts to locate the corresponding order in its internal order book. If no matching order is found, an exception is raised. In this case, two signals are generated: An error message is logged, and the abnormal field for the corresponding instrument in subsequent outputs is set to true. Example log:

If abnormal is true, it is recommended to check the following:

• Whether tick-by-tick data is out of order or missing, which may cause trades or cancellations to arrive before the corresponding order data.

• Whether columns defined in inputColMap contain invalid values, such as missing buy or sell order IDs. This may prevent trade records from matching their original orders.

BSFlag is required for orders and cancellations, as correct order book building depends on it. For trades, it is optional, but still recommended.

If the source tick-by-tick data does not provide BSFlag for either cancellations or trades, it can be inferred as follows:

• Cancellations: set BSFlag to 2 for sell-side cancellations and 1 for buy-side cancellations;

• Trades: set BSFlag to 2 for sell-side aggressive trades and 1 for buy-side aggressive trade trades.

For SZSE data, if cancellation records contain BuyNo and SellNo, where one side is the order ID and the other is 0. Based on the input data preprocessing method in Section 2.2, the BSFlag can be derived as: This logic also applies when BSFlag is N (unknown).

iif(SellNo>BuyNo, 2, 1) as BSFlag

This issue typically has three possible causes:

• Case 1: When exchange is "XSHG", "XSHGSTOCK", "XSHGFUND", "XSHGBOND", or "XSHEBOND", the engine applies internal symbol filtering rules:

  • If exchange= "XSHG" or "XSHGSTOCK", only symbols starting with “6” are output.

  • If exchange= "XSHGFUND", only symbols starting with “5” are output.

  • If exchange= "XSHGBOND" and securitySubType= "ConvertibleBond" (default), only symbols starting with “1” are output.

  • If exchange= "XSHEBOND" and securitySubType= "ConvertibleBond" (default), only symbols starting with “1” are output.

  • As a result, mismatched stocks will produce no output. For example, SZSE stocks (prefix “3”, “0”) will not be processed under exchange = "XSHG". Similarly, symbol formats like "SH.600800" may also be filtered out.

• Case 2: If orderBySeq= true or orderBySeq= (true, ..., ...), missing or non-continuous seqColumn values may cause this issue. Note: For SSE stocks, this parameter defaults to true; for all others, it defaults to false.

  • When orderBySeq= true, no warning is shown in logs.

  • When orderBySeq= (true, ..., ...), abnormal sequence logs will be generated. A sample log is shown below.

  expected seq num: ..., unordered buffer size: ..., the last seq num get: ...

• Case 3: If skipCrossedMarket= true (default in version ≥ 2.00.11.2), snapshots are suppressed when the best bid/ask is crossed (bidPrice1 ≥ askPrice1). From version 2.00.12 onward, corresponding warning logs are added to help with debugging. A sample log is shown below.

  orderBook engine abnormal snapshot.symbol: ... ,ask price ... ,bids price: ... ,timestamp:...);