enableTableCachePurge

Syntax

Details

Enable cache purge for a non-persisted stream table.

Cache purge can be configured using either of the following methods:

• Cache purge by size: Set cacheSize to specify a threshold for the number of records retained. Older records exceeding the threshold will be removed. The threshold is determined as follows:
  • If the number of records appended in one batchdoes not exceed cacheSize, the threshold is 2.5 * cacheSize.
  • If the number of records appended in one batch exceeds cacheSize, the threshold is 1.2 * (appended records + cacheSize).
• Cache purge by time: Set cachePurgeTimeColumn, cachePurgeInterval and cacheRetentionTime. The system will clean up data based on the cachePurgeTimeColumn. Each time when a new record arrives, the system obtains the time difference between the new record and the oldest record kept in memory. If the time difference exceeds cachePurgeInterval, the system will retain only the data with timestamps within cacheRetentionTime of the new data.
• When purging by time, the system uses each batch written in a single bulk write as the purge unit. Data within a batch is not split during purging.Therefore, when a single batch covers a long time range, the data actually retained may cover the time range specified by cacheRetentionTime.If you have strict memory-limit requirements, use cacheSize or reduce the time range covered by each batch.For example, set cachePurgeInterval=40m and cacheRetentionTime=20m. Assume that the data in cachePurgeTimeColumn is sorted in ascending time order, and that the following two batches are written in sequence:

  Batch	Time Range	Time Span
  Batch 1	09:00–09:30	30 minutes
  Batch 2	10:00–10:30	30 minutes

  After batch 1 is written:

  • The timestamp of the first record in memory is 09:00;
  • The latest timestamp of the new data written in this batch is 09:30.
  The system calculates: latest timestamp of the new data (09:30) - timestamp of the first record in memory (09:00) = 30m. Because 30m < cachePurgeInterval (40m), this write does not meet the conditions for triggering a purge, and the system does not perform purging.After batch 2 is written:
  • The first record in memory is still the 09:00 record from batch 1;
  • The latest timestamp of the new data written in this batch is 10:30.
  The system calculates again: latest timestamp of the new data (10:30) - timestamp of the first record in memory (09:00). Because 90m >= cachePurgeInterval (40m), the system uses the latest timestamp of the new data, 10:30, as the reference point and calculates the theoretical retention boundary based on cacheRetentionTime=20m: 10:30 - 20m = 10:10. Under the time-based criterion, data with timestamps earlier than 10:10 is outside the retention range. Therefore:
  • The time range of batch 1 is 09:00-09:30, which is entirely earlier than 10:10, so it is purged;
  • The time range of batch 2 is 10:00-10:30. The data from 10:00 to 10:09 is also earlier than the theoretical retention boundary of 10:10. However, the system uses each batch written in a single bulk write as the purge unit and does not split batch 2 from this write into two parts: 10:00-10:09 and 10:10-10:30. Therefore, batch 2 is retained in full.
  The purge result is as follows:

  Batch	Relationship to Theoretical Retention Boundary	Purge Result
  Batch 1: 09:00-09:30	The entire batch is outside the retention range	Entire batch purged
  Batch 2: 10:00-10:30	The batch spans the theoretical retention boundary	Entire batch retained

Note: If a record has not been enqueued for publishing, it will not be removed.

Parameters

table is an empty stream table.

tableName is a string indicating the name of the shared table.

cacheSize (optional) is a positive integer used to determine the maximum number of records to retain in memory.

cachePurgeTimeColumn (optional) is a STRING scalar indicating the time column in the non-persisted stream table.

cachePurgeInterval (optional) is a DURATION scalar indicating the interval to trigger cache purge.

cacheRetentionTime (optional) is a DURATION scalar indicating the time range used to determine whether data is retained when purging by time. The system uses each batch written in a single bulk write as the purge unit. As a result, the data actually retained may exceed the time range specified by this parameter.

Returns

None.

Examples

Example 1. Set cacheSize.

t = streamTable(1000:0, `time`sym`volume, [DATETIME, SYMBOL, INT])
enableTableCachePurge(table=t, cacheSize=1000)
time = datetime(2024.01.01T09:00:00) +1..1000*2
sym=take(`a`b`c, 1000)
volume = rand(10,1000)

insert into t values([time, sym, volume])
getStreamTableCacheOffset(t)
// output: 0

time = datetime(2024.01.01T09:35:00) +1..1000*2
sym=take(`a`b`c, 1000)
volume = rand(10,1000)
insert into t values([time, sym, volume])
getStreamTableCacheOffset(t)
// output: 500

t = streamTable(1000:0, `time`sym`volume, [DATETIME, SYMBOL, INT])

enableTableCachePurge(table=t, cachePurgeTimeColumn=`time, cachePurgeInterval=30m, cacheRetentionTime=20m)

time = datetime(2024.01.01T09:00:00) +1..1000*2
sym=take(`a`b`c, 1000)
volume = rand(10,1000)

insert into t values([time, sym, volume])
getStreamTableCacheOffset(t)
// output: 0

time = datetime(2024.01.01T09:35:00) +1..1000*2
sym=take(`a`b`c, 1000)
volume = rand(10,1000)
insert into t values([time, sym, volume])
getStreamTableCacheOffset(t)
// output: 999