In-memory Computing

The session is a container. It has an unique ID and stores many defined objects, such as local variables, shared variables, and so on. There are many ways to create a new session, such as launching in command window, XDB connection, GUI connection or Web URL connection. All variables in a session are invisible to other sessions, unless use "share" to explicitly share variables between sessions. Currently DolphinDB only supports sharing table.

When the session is no longer in use, it needs to be closed to release the system resources. If you need to close the XDB session, use command close or set the connection variable to NULL; if you need to close the GUI or Web session, close the GUI window or browser window.

The data in in-memory table is saved in memory. Query and writing into in-memory table is very fast, but the data will be lost after closing the node. It is used to save the temporary data.

There are 2 types of in-memory tables: unpartitioned and partitioned in-memory tables.

All tables created by function table are unpartitioned in-memory tables.

sym = `C`MS`MS`MS`IBM`IBM`C`C`C$symbol
price= 49.6 29.46 29.52 30.02 174.97 175.23 50.76 50.32 51.29
qty = 2200 1900 2100 3200 6800 5400 1300 2500 8800
timestamp = [09:34:07,09:36:42,09:36:51,09:36:59,09:32:47,09:35:26,09:34:16,09:34:26,09:38:12]
t1 = table(timestamp, sym, qty, price);

For more details about unpartitioned tables, please refer to Table.

Working with partitioned in-memory tables can utilize the parallel computing capacities of multi-core CPUs. We can create a partitioned in-memory table by specifying a partitioned in-memory database. Use function database and specify an empty string for database path.

n=10000
month=take(2000.01M..2000.12M, n)
x=rand(1.0, n)
t=table(month, x)

db=database("", VALUE, 2000.01M..2000.12M)
pt = db.createPartitionedTable(t, `pt, `month)
pt.append!(t)
select * from pt;

Use command undef to undefine an in-memory table. For example, to undefine table pt in the example above:

undef(`pt);

A major feature of DolphinDB in-memory computing is that SQL statements can not only manipulate tables, but also other data forms including scalars, vectors, sets, matrices and dictionaries. It significantly increases the flexibility of the scripting language.

In the following example, stock prices are stored in table quotes and stock weights are stored in the dictionary weights. A SQL statement can directly work on the table and the dictionary to calculate the value of the portfolio.

Symbol=take(`AAPL, 6) join take(`FB, 5)
Time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
Price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(Symbol, Time, Price)
weights=dict(`AAPL`FB, 0.6 0.4)
ETF = select Symbol, Time, Price*weights[Symbol] as weightedPrice from quotes
select last(weightedPrice) from ETF pivot by Time, Symbol;

A variable in a session is invisible to other sessions unless it is shared among sessions with statement share . Currently only tables can be shared in DolphinDB. The rows or columns of a shared table cannot be deleted or revised. A shared table can be appended with new data.

share t1 as sharedT1;

In the example above, we share table t1 as sharedT1. In other sessions we can access t1 by visiting sharedT1.

Use command undef to undefine a shared in-memory table:

undef(sharedT1,SHARED)

If we need to frequently conduct concurrent read and write on an in-memory table, but rarely need to update or delete records, DolphinDB provides a special type of in-memory table: MVCC table. An MVCC table records each operation in log files to achieve isolation of read and write. MVCC tables can be persisted to disk and loaded into memory later.

Use function mvccTable to create a multi-version concurrency control table.

n=5
syms=`IBM`C`MS`MSFT`JPM`ORCL`FB`GE
timestamp=09:30:00+rand(18000,n)
sym=rand(syms,n)
qty=100*(1+rand(100,n))
price=5.0+rand(100.0,n)
temp=table(timestamp,sym,qty,price)
t1= mvccTable(1:0,`timestamp`sym`qty`price,[TIMESTAMP,SYMBOL,INT,DOUBLE],"/home/DolphinDB/Data","t1")
t1.append!(temp);

In the example above, the multi-version concurrency control table t1 will be persisted to the disk under path "/home/DolphinDB/Data/t1 ".

Use function loadMvccTable to load the mvcc table into memory.

loadMvccTable("/home/DolphinDB/Data",t1);