In-memory Computing

The session is a container. It has a unique ID and stores many defined objects, such as local variables, shared variables, and so on. There are many ways to create a new session, such as launching in command window, XDB connection, GUI connection or Web URL connection. All variables in a session are invisible to other sessions, unless a share statement is used to explicitly share variables between sessions. Currently DolphinDB only supports sharing tables.

Sessions no longer in use need to be closed to release system resources. To close the XDB session, use command close or set the connection variable to NULL; To close the GUI or Web session, close the GUI window or browser window.

In-memory tables store data directly within the system's memory so that the data can be queried and written into memory at a fast speed. However, the data will be lost when the node is closed. Consequently, in-memory tables are typically used to save temporary data.

There are 2 types of in-memory tables: unpartitioned and partitioned in-memory tables.

All tables created by function table are unpartitioned in-memory tables.

sym = `C`MS`MS`MS`IBM`IBM`C`C`C$symbol
price= 49.6 29.46 29.52 30.02 174.97 175.23 50.76 50.32 51.29
qty = 2200 1900 2100 3200 6800 5400 1300 2500 8800
timestamp = [09:34:07,09:36:42,09:36:51,09:36:59,09:32:47,09:35:26,09:34:16,09:34:26,09:38:12]
t1 = table(timestamp, sym, qty, price);

For details about unpartitioned in-memory tables, please refer to Table.

Partitioned in-memory tables can utilize the parallel computing capacities of multi-core CPUs. To create a partitioned in-memory table, you need to specify a partitioned in-memory database. Use function database and specify an empty string for database directory.

n=10000
month=take(2000.01M..2000.12M, n)
x=rand(1.0, n)
t=table(month, x)

db=database("", VALUE, 2000.01M..2000.12M)
pt = db.createPartitionedTable(t, `pt, `month)
pt.append!(t)
select * from pt;

Use command undef to undefine an in-memory table. For example, to undefine table "pt" in the example above, use the following script:

undef(`pt);

In the following example, stock prices are stored in table "quotes" and stock weights are stored in dictionary "weights". The value of the portfolio can be calculated with only one SQL statement based on the table and the dictionary.

Symbol=take(`AAPL, 6) join take(`FB, 5)
Time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
Price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(Symbol, Time, Price)
weights=dict(`AAPL`FB, 0.6 0.4)
ETF = select Symbol, Time, Price*weights[Symbol] as weightedPrice from quotes
select last(weightedPrice) from ETF pivot by Time, Symbol;

A variable in a session is invisible to other sessions unless it is shared among sessions with statement share. Currently only tables can be shared in DolphinDB. A shared table can be queried and appended with new data, but its rows or columns cannot be deleted or revised.

share t1 as sharedT1;

In the example above, we share table "t1" as "sharedT1". In other sessions we can access "t1" by visiting "sharedT1".

Use command undef to undefine a shared in-memory table, as shown in the following script:

undef(sharedT1,SHARED)

MVCC table is a special type of in-memory table provided by DolphinDB for the case when concurrent read and write operations are frequently conducted in memory while records are rarely updated or deleted. An MVCC table records each operation in log files to achieve isolation of read and write. It can also be persisted to disk and loaded into memory for subsequent operations.

Use function mvccTable to create a multi-version concurrency control table.

n=5
syms=`IBM`C`MS`MSFT`JPM`ORCL`FB`GE
timestamp=09:30:00+rand(18000,n)
sym=rand(syms,n)
qty=100*(1+rand(100,n))
price=5.0+rand(100.0,n)
temp=table(timestamp,sym,qty,price)
t1= mvccTable(1:0,`timestamp`sym`qty`price,[TIMESTAMP,SYMBOL,INT,DOUBLE],"/home/DolphinDB/Data","t1")
t1.append!(temp);

In the example above, the multi-version concurrency control table "t1" will be persisted to the disk under path "/home/DolphinDB/Data/t1 ".

Use function loadMvccTable to load the mvcc table into memory.

loadMvccTable("/home/DolphinDB/Data",t1);