Data Model

DolphinDB is a columnar database. DolphinDB partitions large-scale data sets horizontally according to certain rules that may be specified by the users.

The following example intuitively explain the storage model of DolphinDB. First, create the database and table:

db=database("dfs://db1", VALUE, 1 2 3 4)
id=take(1..4, 1000)
x=take(1.0, 1000)
t=table(id, x)
pt=db.createPartitionedTable(t,`pt,`id)
pt.append!(t);

The partitions in the example above are based on the values of column id. All recoreds with id=1 are stored in a partition; all records with id=2 are stored in another partition, etc. Each partition has 2 files (id.col and x.col), each of which indicating a column.

If the copy number is 2, the table pt totally has 2*4*2=16 column files.

Other than the value partitions used in this example, DolphinDB also supports range, list, hash and composite partitions. For details please refer to the section of Create Databases and Tables.

DolphinDB supports lossless compression. It uses the LZ4 compression algorithm that offers fast compression speed and decent compression ratio. For financial data, the compression ratio can reach 20% to 25%. If there are more duplicate values in the same column, the compression ratio will be higher. DolphinDB conducts compression in each newly added batch, so large batch writes generally have higher compression ratio than small batch writes. If each time only one record is added to the database, then the new data is not compressed at all (if cache engine is not enabled).

When reading data, the system fetchs the required columns from disk, decompresses them and loads them into memory.

Modern operating systems provide page buffering to improve I/O performance. When data is written to the database, it is written to the buffer page of the operating system first instead of directly to disk. A system crash or power outage may result in data loss. DolphinDB provides 2 mechanisms to write database logs (including redo log and metadata edit log): either the log files must be written to disk before each transaction is committed, or the log files are written to cache before a transaction is committed and the operating system will decide when to write the log files to disk at a later time. The first strategy (by setting dataSync=1) guarantees no data loss due to system crash or power outage at the cost of slower write speed. A cost-effective strategy is to write the database log files to a low-capacity but high-performance SSD and massive amounts of data to large-capacity HDDs.

A large number of small batches of writes have a negative effect on disk I/O performance and data compression ratio. Considering this, DolphinDB provides the configuration parameter OLAPCacheEngineSizethat specifies the capacity of cache engine. If cache engine is enabled, data is not written to disk until data in cache exceeds a threshold. To enable the cache engine, set chunkCacheEngineMemSize>0 and dataSync=1.

DolphinDB is primarily designed for storage, retrieval, analysis and computation of huge volumes of structured data. As of now, it does not support the modification and deletion of single records. Modification and deletion of data must be conducted at the partitions level. The database in the example above uses a value domain with id as the partitioning column. Regarding the partition of id=1 in table pt, we can only delete all records with id=1 but not any particular record with id=1. To modify a record with id=1, we need to load the entire partition with id=1 into memory, modify it, then delete all previous records in the partition and save the modified partition to table pt. Please refer to the section of DropPartitions for details.