pivot by
pivot by is a unique feature in DolphinDB and an extension to the ANSI
            SQL. It rearranges a column (or multiple columns) of a table (with or without a data
            transformation function) on two dimensions. If the pivot by clause
            includes more than 2 columns, the last column determines the columns in the result and
            all other columns determine the rows in the result. It produces a table with
                select statement and a matrix with exec statement.
            When used in conjunction with the pivot by clause, the select clause can specify
            multiple columns. See pivot and
            unpivot .
The following versions contain enhancements for the pivot by statement:
| Version | Enhancements | 
|---|---|
| 1.20.4 | 1. If the pivot by clause includes more than 2 columns, the last column determines the columns in the result and all other columns determine the rows in the result. 2. When used in conjunction with the pivot by clause, the select clause can specify multiple columns.  | 
                    
Examples
Example 1
sym = `C`MS`MS`MS`IBM`IBM`C`C`C
price= 49.6 29.46 29.52 30.02 174.97 175.23 50.76 50.32 51.29
qty = 2200 1900 2100 3200 6800 5400 1300 2500 8800
timestamp = [09:34:07,09:35:42,09:36:51,09:36:59,09:35:47,09:36:26,09:34:16,09:35:26,09:36:12]
t2 = table(timestamp, sym, qty, price);
t2;
            | timestamp | sym | qty | price | 
|---|---|---|---|
| 09:34:07 | C | 2200 | 49.6 | 
| 09:35:42 | MS | 1900 | 29.46 | 
| 09:36:51 | MS | 2100 | 29.52 | 
| 09:36:59 | MS | 3200 | 30.02 | 
| 09:35:47 | IBM | 6800 | 174.97 | 
| 09:36:26 | IBM | 5400 | 175.23 | 
| 09:34:16 | C | 1300 | 50.76 | 
| 09:35:26 | C | 2500 | 50.32 | 
| 09:36:12 | C | 8800 | 51.29 | 
select price from t2 pivot by timestamp, sym;
            | timestamp | C | IBM | MS | 
|---|---|---|---|
| 09:34:07 | 49.6 | ||
| 09:34:16 | 50.76 | ||
| 09:35:26 | 50.32 | ||
| 09:35:42 | 29.46 | ||
| 09:35:47 | 174.97 | ||
| 09:36:12 | 51.29 | ||
| 09:36:26 | 175.23 | ||
| 09:36:51 | 29.52 | ||
| 09:36:59 | 30.02 | 
a=select last(price) from t2 pivot by timestamp.minute(), sym;
a;
            | minute_timestamp | C | IBM | MS | 
|---|---|---|---|
| 09:34m | 50.76 | ||
| 09:35m | 50.32 | 174.97 | 29.46 | 
| 09:36m | 51.29 | 175.23 | 30.02 | 
typestr a;
// output
TABLE
b=exec count(price) from t2 pivot by timestamp.minute(), sym;
b;
            | label | C | IBM | MS | 
|---|---|---|---|
| 09:34m | 2 | ||
| 09:35m | 1 | 1 | 1 | 
| 09:36m | 1 | 1 | 2 | 
typestr b;
// output
FAST DOUBLE MATRIX
            Example 2. Calculate the basket value of an ETF's constituents. For simplicity, assume an ETF has 2 constituents AAPL and FB.
symbol=take(`AAPL, 6) join take(`FB, 5)
time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(symbol, time, price)
weights=dict(`AAPL`FB, 0.6 0.4)
ETF = select symbol, time, price*weights[symbol] as price from quotes;
select price from ETF pivot by time, symbol;
            | Time | AAPL | FB | 
|---|---|---|
| 2019.02.27T09:45:01.000000146 | 103.962 | |
| 2019.02.27T09:45:01.000000212 | 64.604 | |
| 2019.02.27T09:45:01.000000278 | 103.956 | |
| 2019.02.27T09:45:01.000000412 | 103.944 | |
| 2019.02.27T09:45:01.000000445 | 103.95 | |
| 2019.02.27T09:45:01.000000496 | 103.956 | |
| 2019.02.27T09:45:01.000000556 | 64.6 | |
| 2019.02.27T09:45:01.000000598 | 64.596 | |
| 2019.02.27T09:45:01.000000712 | 64.6 | |
| 2019.02.27T09:45:01.000000789 | 103.962 | |
| 2019.02.27T09:45:01.000000989 | 64.604 | 
To calculate the basket value of this ETF's constituents at each timestamp that any of its constituents had a trade, we need to forward fill each stock's price in the pivot table, and then sum each row's weighted price. They can be completed in just one SQL statement with "pivot by".
select rowSum(ffill(price)) from ETF pivot by time, symbol;
            | Time | rowSum | 
|---|---|
| 2019.02.27T09:45:01.000000146 | 103.962 | 
| 2019.02.27T09:45:01.000000212 | 168.566 | 
| 2019.02.27T09:45:01.000000278 | 168.56 | 
| 2019.02.27T09:45:01.000000412 | 168.548 | 
| 2019.02.27T09:45:01.000000445 | 168.554 | 
| 2019.02.27T09:45:01.000000496 | 168.56 | 
| 2019.02.27T09:45:01.000000556 | 168.556 | 
| 2019.02.27T09:45:01.000000598 | 168.552 | 
| 2019.02.27T09:45:01.000000712 | 168.556 | 
| 2019.02.27T09:45:01.000000789 | 168.562 | 
| 2019.02.27T09:45:01.000000989 | 168.566 | 
Example 3. The names of the newly created columns from a pivot by clause will be automatically adjusted.
Before version 1.30.14, column names in DolphinDB can only use letters, digits or underscores (_), and must start with letters.
Since version 1.30.14, column names generated by pivot by, addColumn can contain special characters or start with digits.
For details, please refer to section Creating tables in Table.
date = take(2021.08.01  2021.08.02 2021.08.03, 12)
sym = take(["IBM N", "_MSFTN", "3_GOOGS", ""], 12).sort()
value = 1..12
t=table(date, sym, value)
re = select value from t pivot by date, sym
            | date | NULL | 3_GOOGS | IBM N | _MSFTN | 
|---|---|---|---|---|
| 2021.08.01 | 1 | 4 | 7 | 10 | 
| 2021.08.02 | 2 | 5 | 8 | 11 | 
| 2021.08.03 | 3 | 6 | 9 | 12 | 
select  _"NULL" from re
            | NULL | 
|---|
| 1 | 
| 2 | 
| 3 | 
select  _"3_GOOGS" from re
            | 3_GOOGS | 
|---|
| 4 | 
| 5 | 
| 6 | 
In the following example, set removeSpecialCharInColumnName = true. The stock symbols "600300" and "600600" are adjusted to "c600300" and "c600600" as column names.
symbol=take(`600300, 6) join take(`600600, 5)
time=2019.02.27T09:45:01.000000000+[146, 278, 412, 445, 496, 789, 212, 556, 598, 712, 989]
price=173.27 173.26 173.24 173.25 173.26 173.27 161.51 161.50 161.49 161.50 161.51
quotes=table(symbol, time, price)
weights=dict(`600300`600600, 0.6 0.4)
ETF = select symbol, time, price*weights[symbol] as price from quotes;
select last(price) from ETF pivot by time, symbol;
            | time | C600300 | C600600 | 
|---|---|---|
| 2019.02.27T09:45:01.000000146 | 103.962 | |
| 2019.02.27T09:45:01.000000212 | 64.604 | |
| 2019.02.27T09:45:01.000000278 | 103.956 | |
| 2019.02.27T09:45:01.000000412 | 103.944 | |
| 2019.02.27T09:45:01.000000445 | 103.95 | |
| 2019.02.27T09:45:01.000000496 | 103.956 | |
| 2019.02.27T09:45:01.000000556 | 64.6 | |
| 2019.02.27T09:45:01.000000598 | 64.596 | |
| 2019.02.27T09:45:01.000000712 | 64.6 | |
| 2019.02.27T09:45:01.000000789 | 103.962 | |
| 2019.02.27T09:45:01.000000989 | 64.604 | 
Example 4. The pivot by clause includes more than 2 columns.
date = 2020.09.21 + 0 0 0 0 1 1 1 1
sym = `MS`MS`GS`GS`MS`MS`GS`GS$SYMBOL
factorNum = 1 2 1 2 1 2 1 2
factorValue = 1.2 -3.4 -2.5 6.3 1.1 -3.2 -2.1 5.6
t = table(date, sym, factorNum, factorValue);
t;
            | date | sym | factorNum | factorValue | 
|---|---|---|---|
| 2020.09.21 | MS | 1 | 1.2 | 
| 2020.09.21 | MS | 2 | -3.4 | 
| 2020.09.21 | GS | 1 | -2.5 | 
| 2020.09.21 | GS | 2 | 6.3 | 
| 2020.09.22 | MS | 1 | 1.1 | 
| 2020.09.22 | MS | 2 | -3.2 | 
| 2020.09.22 | GS | 1 | -2.1 | 
| 2020.09.22 | GS | 2 | 5.6 | 
select factorValue from t pivot by date, sym, factorNum;
            | date | sym | 1 | 2 | 
|---|---|---|---|
| 2020.09.21 | GS | -2.5 | 6.3 | 
| 2020.09.21 | MS | 1.2 | -3.4 | 
| 2020.09.22 | GS | -2.1 | 5.6 | 
| 2020.09.22 | MS | 1.1 | -3.2 | 
Select multiple columns for each factorNum
select factorValue, factorValue>0 as factorSign from t pivot by date, sym, factorNum;
            | date | sym | factorValue_1 | factorValue_2 | factorSign_1 | factorSign_2 | 
|---|---|---|---|---|---|
| 2020.09.21 | GS | -2.5 | 6.3 | 0 | 1 | 
| 2020.09.21 | MS | 1.2 | -3.4 | 1 | 0 | 
| 2020.09.22 | GS | -2.1 | 5.6 | 0 | 1 | 
| 2020.09.22 | MS | 1.1 | -3.2 | 1 | 0 | 
