Higher-Order Functions
Introduction
A template is a built-in higher order function, which can extend or enhance a function or an operator. A template takes a function and some objects as input. It works like a pipe between the function and its input data. In general, the input data is dissembled into multiple pieces (which may or may not overlap) in a preset way at first; then the individual pieces of data are applied to the given function to produce results one at a time; finally all individual results are assembled into one object to return. The input data for a template can be vectors, matrices or tables, with the occasional use of scalars and dictionaries. With a template, complicated analytical tasks can be accomplished very efficiently with only a few lines of statements.
Syntax
Higher-order functions are always used together with operators, user-defined functions, or system functions. Higher-order function symbols start with the symbol colon ":" followed by an upper-case single character.
higherOrderFunctionName (<functionName>, functionParameter1, ...functionParameterN)
or
<functionName> :<higher order symbol> functionParameter
or
functionParameter1 <functionName> :<higher order symbol> functionParameter2
In DolphinDB, the first parameter of all templates must be a function name. To call
more than one function in a higher-order function, you can specify the call
higher-order function as the first parameter <functionName>, and then pass
in the functions you want to call as argument to call. Please refer
to the section about the higher-order function call.
Higher-order Function Summary
The summary table below shows template names and applications.
| Symbol | Name | Applications | Examples |
|---|---|---|---|
| :E | each | unary operators, binary operators, function calls | |
| peach | parallel version of each |
||
| :R | eachRight | binary operators | |
| :L | eachLeft | binary operators | |
| :P | eachPre | binary operators, function calls | |
| :O | eachPost | binary operators, function calls | |
| pivot | function calls | Transpose rows and columns of raw data or grouped aggregation results | |
| :A | accumulate | binary operators, function calls | |
| :T | reduce | binary operators, function calls | |
| :G | groupby | binary operators, function calls | |
| :C | cross | binary operators, function calls | |
| pcross | parallel version of cross |
Work with pivoting | |
| moving | binary operators, function calls (aggregate function) | ||
| window | function calls | Similar to moving. Compared with
the moving function, the window function has a more flexible window.
The left/right boundary of the window specified in the
window function can be both inclusive and
exclusive. |
|
| nullCompare | binary operators, function calls | Preserve NULLs in the calculation to keep consistent with the behavior of Python. | |
| loop | unary operators, binary operators, function calls, mixed return types | Import text files | |
| ploop | parallel version of loop |
||
| all | binary operators, function calls | ||
| any | binary operators, function calls | ||
| call | function calls | Usually used with function each to
simultaneously call a batch of functions |
|
| pcall | parallel version of call |
||
| unifiedCall | function calls | Similar to call. The size of
args in unifiedCall is always 1. |
|
| :X | contextby | binary operators, function calls | Perform specified calculations in groups |
| segmentby | binary operators, function calls | ||
| rolling | binary operators, function calls | Calculate the beta value of APPL on the market (SPY) | |
| withNullFill | binary operators | Replace the NULL value with a specific value in the calculation | |
| byRow | function calls | Apply the function to each row of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple. | |
| byColumn | function calls | Apply the function to each column of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple. | |
| talib | function calls | To process data with DolphinDB functions in the same way as Python TA-lib | |
| tmoving | function calls | Apply the function/operator to a time-based sliding window (with the right boundary inclusive) of the given objects. | |
| twindow | function calls | Similar to tmoving. Compared with
the tmoving function, twindow has more flexible
windows. The left/right boundary of the window specified in the
twindow function can be both inclusive and
exclusive. |
The adverbs listed above can be used iteratively.
The operations of each adverb are conducted in left-to-right order. Take X <operator> :E:L Y for example, the leftmost adverb :E is first applied to each element in X and Y (X(i) and Y(i)), and then the next adverb :L is used to apply the operator for each X(i). The result is returned after all adverbs are applied.
a=1 2 3
b=4 5 6
c=(a,b)
re=c +:E:L c
re
// output
(1 2 3
- - -
2 3 4
3 4 5
4 5 6
,4 5 6
-- -- --
8 9 10
9 10 11
10 11 12
)Rules to Dissemble and Assemble
In general, a vector is dissembled into scalars, a matrix into columns (vectors), a tuple into multiple elements (of different data forms), and a table into rows (represented by dictionaries).
In the assembly phase, the results of scalar type are merged to form a vector, vectors to a matrix, dictionaries to a table, and other data forms (which are inconsistent) into a tuple.
A template function iterates over a vector/tuple by elements, a matrix by columns, and a table by rows.