setRetentionPolicy

setRetentionPolicy(dbHandle, retentionHours, [retentionDimension], [hoursToColdVolume])

Set the policy of data retention and tiered storage. The parameter retentionHours should be specified as large as possible for tiered storage to avoid deleting any data.

For databases whose partitioning scheme includes the DATE or DATEHOUR type:

Both data retention and tiered storage are partition-based. Therefore, the interval configured by retentionHours and hoursToColdVolume must be divisible by the granularity of the partition.

The system will keep the data with the timestamp of the temporal partitioning column within the last retentionHours based on the system time: Data of the last hoursToColdVolume are stored in volumes; Data in [current time - hoursToColdVolumes - 10 days, current time - hoursToColdVolumes) are migrated to coldVolumes. If multiple paths are specified for coldVolumes, the data will be transferred randomly to one of the specified directories.

For other data, only data in the range of [current time - retentionHours - 10 days, current time - retentionHours) are deleted. To delete the data outside the range, you can call function dropPartition.

For databases whose partitioning scheme includes the MONTH type:

Tiered storage and data retention policies still operate at the partition level. For a monthly partition M, when the current time - hoursToColdVolume falls into the next calendar month M+1 or later, it will be automatically migrated to coldVolumes; when the current time - retentionHours falls into the next calendar month M+1 or later, it will be deleted.

dbHandle is a database handle. The data type of at least one of the partitioning columns of the database must be DATE, DATEHOUR or MONTH.

retentionHours is a positive integer indicating the number of hours that data are kept.

retentionDimension (optional) is an integer indicating the layer of the temporal partition. The default value is 0 indicating the first layer of a COMPO partition is a temporal partition.

hoursToColdVolume (optional) is a positive integer indicating the number of hours that data are kept in volumes. Data stored in volumes will be migrated to the specified coldVolumes (configuration parameter) after hoursToColdVolume.

db=database("dfs://db1",VALUE,2019.06.01..date(now()))
retentionHour=9*24
hoursToColdVolume=1*24
setRetentionPolicy(db,retentionHour,0, hoursToColdVolume);

schema(db);
/* output:
partitionSchema->[2022.05.05,2022.05.04,2022.05.03,2022.05.02,2022.05.01,2022.04.30,2022.04.29,2022.04.28,2022.04.27,2022.04.26,...]
partitionSites->
partitionTypeName->VALUE
hoursToColdVolume->24
atomic->TRANS
databaseDir->dfs://db1
engineType->OLAP
chunkGranularity->TABLE
retentionDimension->0
partitionType->1
retentionHours->216
*/

Configure tiered storage for a database partitioned by month. To enable tiered storage, configure the storage path for cold data first. See Reference for details.

db = database("dfs://monthlyDB", VALUE, 2000.01M..2025.12M)
retentionHour=31*24
hoursToColdVolume=5*24
setRetentionPolicy(db,retentionHour,0, hoursToColdVolume);