haStreamTable

Syntax

haStreamTable(raftGroup, table, tableName, cacheLimit, [keyColumn], [retentionMinutes=1440])

Arguments

raftGroup is an integer greater than 1 indicating Raft group ID.

table is an empty table object created by function table.

tableName is a string indicating the name of the high-availability stream table.

cacheLimit is an integer representing the maximum number of rows of the high-availability stream table to be kept in memory. If cacheLimit>100,000, it is automatically adjusted to 100,000.

keyColumn (optional) is a string indicating the name of the primary key column.

retentionMinutes (optional) is an integer indicating for how long (in terms of minutes) a log file larger than 1GB will be kept after last update. The default value is 1440, which means the log file only keeps data in the past 24 hours.

Details

Create a high-availability stream table. To use the function, we must enable high availability for streaming by specifying parameters streamingHAMode and streamingRaftGroups in cluster configuration file cluster.cfg.

As the cluster starts up, the data nodes specified by the configuration parameter streamingRaftGroups form Raft groups. In a Raft group, one data node is the Leader and the rest are Followers. There is a copy of the high-availability stream table on each data node in a Raft group.

After creating the high-availability stream table, subscribe to the high-availability stream table on any of the data nodes in a Raft group and set parameter reconnect of function subscribeTable to "true". The high-availability stream table on the Leader will publish data. If the Leader node goes down, the system will elect a new Leader to continue publishing data. Subscribers will automatically be connected to the high-availability stream table on the new Leader.

A Raft group can have multiple high-availability streaming tables.

Examples

Suppose streamingRaftGroups=11:NODE1:NODE2:NODE3. Execute the following script on any data node of the Raft group to create a high-availability stream table trades:

colNames = `timestamp`sym`qty`price
colTypes = [TIMESTAMP,SYMBOL,INT,DOUBLE]
t=table(1:0,colNames,colTypes)
haStreamTable(11,t,`trades,100000);

Execute the followng script on another node that do not belong to the Raft group (NODE4) to subscribe to table trades, and then save the subscribed data to a distributed database.

if(existsDatabase("dfs://stock")){
   dropDatabase("dfs://stock")
}
db=database("dfs://stock",VALUE,2018.08.01..2019.12.30)
t=table(1:0,`timestamp`sym`qty`price,[TIMESTAMP,SYMBOL,INT,DOUBLE])
trades_slave=db.createPartitionedTable(t,`trades_slave,`timestamp);
subscribeTable(NODE2,`trades,`sub_trades,-1,append!{trades_slave},true,1000,1,-1,true);

Please note that in the script above, the first parameter of function subscribeTable can be any of NODE1, NODE2 and NODE3. Paramater reconnect must be set to "true".

Execute the following script on NODE4 to cancel the subscription.

unsubscribeTable(NODE2,`trades,`sub_trades);