pivot by
pivot by
is a unique feature in DolphinDB and an extension to the ANSI
SQL. It rearranges a column (or multiple columns) of a table (with or without a data
transformation function) on two dimensions. If the pivot by
clause
includes more than 2 columns, the last column determines the columns in the result and
all other columns determine the rows in the result. It produces a table with
select
statement and a matrix with exec
statement.
When used in conjunction with the pivot by clause, the select clause can specify
multiple columns. See pivot and
unpivot .
The following versions contain enhancements for the pivot by statement:
Version | Enhancements |
---|---|
1.20.4 | 1. If the pivot by clause includes more than 2 columns, the last column determines the columns in the result and all other columns determine the rows in the result. 2. When used in conjunction with the pivot by clause, the select clause can specify multiple columns. |
Examples
Example 1
sym = `C`MS`MS`MS`IBM`IBM`C`C`C
price= 49.6 29.46 29.52 30.02 174.97 175.23 50.76 50.32 51.29
qty = 2200 1900 2100 3200 6800 5400 1300 2500 8800
timestamp = [09:34:07,09:35:42,09:36:51,09:36:59,09:35:47,09:36:26,09:34:16,09:35:26,09:36:12]
t2 = table(timestamp, sym, qty, price);
t2;
timestamp | sym | qty | price |
---|---|---|---|
09:34:07 | C | 2200 | 49.6 |
09:35:42 | MS | 1900 | 29.46 |
09:36:51 | MS | 2100 | 29.52 |
09:36:59 | MS | 3200 | 30.02 |
09:35:47 | IBM | 6800 | 174.97 |
09:36:26 | IBM | 5400 | 175.23 |
09:34:16 | C | 1300 | 50.76 |
09:35:26 | C | 2500 | 50.32 |
09:36:12 | C | 8800 | 51.29 |
select price from t2 pivot by timestamp, sym;
timestamp | C | IBM | MS |
---|---|---|---|
09:34:07 | 49.6 | ||
09:34:16 | 50.76 | ||
09:35:26 | 50.32 | ||
09:35:42 | 29.46 | ||
09:35:47 | 174.97 | ||
09:36:12 | 51.29 | ||
09:36:26 | 175.23 | ||
09:36:51 | 29.52 | ||
09:36:59 | 30.02 |
a=select last(price) from t2 pivot by timestamp.minute(), sym;
a;
minute_timestamp | C | IBM | MS |
---|---|---|---|
09:34m | 50.76 | ||
09:35m | 50.32 | 174.97 | 29.46 |
09:36m | 51.29 | 175.23 | 30.02 |
typestr a;
// output
TABLE
b=exec count(price) from t2 pivot by timestamp.minute(), sym;
b;
label | C | IBM | MS |
---|---|---|---|
09:34m | 2 | ||
09:35m | 1 | 1 | 1 |
09:36m | 1 | 1 | 2 |
typestr b;
// output
FAST DOUBLE MATRIX
Example 2. Calculate the basket value of an ETF's constituents. For simplicity, assume an ETF has 2 constituents AAPL and FB.
symbol=take(`AAPL, 6) join take(`FB, 5)
time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(symbol, time, price)
weights=dict(`AAPL`FB, 0.6 0.4)
ETF = select symbol, time, price*weights[symbol] as price from quotes;
select price from ETF pivot by time, symbol;
Time | AAPL | FB |
---|---|---|
2019.02.27T09:45:01.000000146 | 103.962 | |
2019.02.27T09:45:01.000000212 | 64.604 | |
2019.02.27T09:45:01.000000278 | 103.956 | |
2019.02.27T09:45:01.000000412 | 103.944 | |
2019.02.27T09:45:01.000000445 | 103.95 | |
2019.02.27T09:45:01.000000496 | 103.956 | |
2019.02.27T09:45:01.000000556 | 64.6 | |
2019.02.27T09:45:01.000000598 | 64.596 | |
2019.02.27T09:45:01.000000712 | 64.6 | |
2019.02.27T09:45:01.000000789 | 103.962 | |
2019.02.27T09:45:01.000000989 | 64.604 |
To calculate the basket value of this ETF's constituents at each timestamp that any of its constituents had a trade, we need to forward fill each stock's price in the pivot table, and then sum each row's weighted price. They can be completed in just one SQL statement with "pivot by".
select rowSum(ffill(price)) from ETF pivot by time, symbol;
Time | rowSum |
---|---|
2019.02.27T09:45:01.000000146 | 103.962 |
2019.02.27T09:45:01.000000212 | 168.566 |
2019.02.27T09:45:01.000000278 | 168.56 |
2019.02.27T09:45:01.000000412 | 168.548 |
2019.02.27T09:45:01.000000445 | 168.554 |
2019.02.27T09:45:01.000000496 | 168.56 |
2019.02.27T09:45:01.000000556 | 168.556 |
2019.02.27T09:45:01.000000598 | 168.552 |
2019.02.27T09:45:01.000000712 | 168.556 |
2019.02.27T09:45:01.000000789 | 168.562 |
2019.02.27T09:45:01.000000989 | 168.566 |
Example 3. The names of the newly created columns from a pivot by clause will be automatically adjusted.
Before version 1.30.14, column names in DolphinDB can only use letters, digits or underscores (_), and must start with letters.
Since version 1.30.14, column names generated by pivot by, addColumn can contain special characters or start with digits.
For details, please refer to section Creating tables in Table.
date = take(2021.08.01 2021.08.02 2021.08.03, 12)
sym = take(["IBM N", "_MSFTN", "3_GOOGS", ""], 12).sort()
value = 1..12
t=table(date, sym, value)
re = select value from t pivot by date, sym
date | NULL | 3_GOOGS | IBM N | _MSFTN |
---|---|---|---|---|
2021.08.01 | 1 | 4 | 7 | 10 |
2021.08.02 | 2 | 5 | 8 | 11 |
2021.08.03 | 3 | 6 | 9 | 12 |
select _"NULL" from re
NULL |
---|
1 |
2 |
3 |
select _"3_GOOGS" from re
3_GOOGS |
---|
4 |
5 |
6 |
In the following example, set removeSpecialCharInColumnName = true. The stock symbols "600300" and "600600" are adjusted to "c600300" and "c600600" as column names.
symbol=take(`600300, 6) join take(`600600, 5)
time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(symbol, time, price)
weights=dict(`600300`600600, 0.6 0.4)
ETF = select symbol, time, price*weights[symbol] as price from quotes;
select last(price) from ETF pivot by time, symbol;
time | C600300 | C600600 |
---|---|---|
2019.02.27T09:45:01.000000146 | 103.962 | |
2019.02.27T09:45:01.000000212 | 64.604 | |
2019.02.27T09:45:01.000000278 | 103.956 | |
2019.02.27T09:45:01.000000412 | 103.944 | |
2019.02.27T09:45:01.000000445 | 103.95 | |
2019.02.27T09:45:01.000000496 | 103.956 | |
2019.02.27T09:45:01.000000556 | 64.6 | |
2019.02.27T09:45:01.000000598 | 64.596 | |
2019.02.27T09:45:01.000000712 | 64.6 | |
2019.02.27T09:45:01.000000789 | 103.962 | |
2019.02.27T09:45:01.000000989 | 64.604 |
Example 4. The pivot by clause includes more than 2 columns.
date = 2020.09.21 + 0 0 0 0 1 1 1 1
sym = `MS`MS`GS`GS`MS`MS`GS`GS$SYMBOL
factorNum = 1 2 1 2 1 2 1 2
factorValue = 1.2 -3.4 -2.5 6.3 1.1 -3.2 -2.1 5.6
t = table(date, sym, factorNum, factorValue);
t;
date | sym | factorNum | factorValue |
---|---|---|---|
2020.09.21 | MS | 1 | 1.2 |
2020.09.21 | MS | 2 | -3.4 |
2020.09.21 | GS | 1 | -2.5 |
2020.09.21 | GS | 2 | 6.3 |
2020.09.22 | MS | 1 | 1.1 |
2020.09.22 | MS | 2 | -3.2 |
2020.09.22 | GS | 1 | -2.1 |
2020.09.22 | GS | 2 | 5.6 |
select factorValue from t pivot by date, sym, factorNum;
date | sym | 1 | 2 |
---|---|---|---|
2020.09.21 | GS | -2.5 | 6.3 |
2020.09.21 | MS | 1.2 | -3.4 |
2020.09.22 | GS | -2.1 | 5.6 |
2020.09.22 | MS | 1.1 | -3.2 |
Select multiple columns for each factorNum
select factorValue, factorValue>0 as factorSign from t pivot by date, sym, factorNum;
date | sym | factorValue_1 | factorValue_2 | factorSign_1 | factorSign_2 |
---|---|---|---|---|---|
2020.09.21 | GS | -2.5 | 6.3 | 0 | 1 |
2020.09.21 | MS | 1.2 | -3.4 | 1 | 0 |
2020.09.22 | GS | -2.1 | 5.6 | 0 | 1 |
2020.09.22 | MS | 1.1 | -3.2 | 1 | 0 |