Higher-Order Functions

Introduction

A template is a built-in higher order function, which can extend or enhance a function or an operator. A template takes a function and some objects as input. It works like a pipe between the function and its input data. In general, the input data is dissembled into multiple pieces (which may or may not overlap) in a preset way at first; then the individual pieces of data are applied to the given function to produce results one at a time; finally all individual results are assembled into one object to return. The input data for a template can be vectors, matrices or tables, with the occasional use of scalars and dictionaries. With a template, complicated analytical tasks can be accomplished very efficiently with only a few lines of statements.

Syntax

Higher-order functions are always used together with operators, user-defined functions, or system functions. Higher-order function symbols start with the symbol colon ":" followed by an upper-case single character.

higherOrderFunctionName (<functionName>, functionParameter1, ...functionParameterN)

or

<functionName> :<higher order symbol> functionParameter

or

functionParameter1 <functionName> :<higher order symbol> functionParameter2

In DolphinDB, the first parameter of all templates must be a function name. To call more than one function in a higher-order function, you can specify the call higher-order function as the first parameter <functionName>, and then pass in the functions you want to call as argument to call. Please refer to the section about the higher-order function call.

Higher-order Function Summary

The summary table below shows template names and applications.

Symbol Name Applications Examples
:E each unary operators, binary operators, function calls
peach parallel version of each
:R eachRight binary operators
:L eachLeft binary operators
:P eachPre binary operators, function calls
:O eachPost binary operators, function calls
pivot function calls Transpose rows and columns of raw data or grouped aggregation results
:A accumulate binary operators, function calls
:T reduce binary operators, function calls
:G groupby binary operators, function calls
:C cross binary operators, function calls
pcross parallel version of cross Work with pivoting
moving binary operators, function calls (aggregate function)
window function calls Similar to moving. Compared with the moving function, the window function has a more flexible window. The left/right boundary of the window specified in the window function can be both inclusive and exclusive.
nullCompare binary operators, function calls Preserve NULLs in the calculation to keep consistent with the behavior of Python.
loop unary operators, binary operators, function calls, mixed return types Import text files
ploop parallel version of loop
all binary operators, function calls
any binary operators, function calls
call function calls Usually used with function each to simultaneously call a batch of functions
pcall parallel version of call
unifiedCall function calls Similar to call. The size of args in unifiedCall is always 1.
:X contextby binary operators, function calls Perform specified calculations in groups
segmentby binary operators, function calls
rolling binary operators, function calls Calculate the beta value of APPL on the market (SPY)
withNullFill binary operators Replace the NULL value with a specific value in the calculation
:H byRow function calls Apply the function to each row of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple.
:V byColumn function calls Apply the function to each column of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple.
talib function calls To process data with DolphinDB functions in the same way as Python TA-lib
tmoving function calls Apply the function/operator to a time-based sliding window (with the right boundary inclusive) of the given objects.
twindow function calls Similar to tmoving. Compared with the tmoving function, twindow has more flexible windows. The left/right boundary of the window specified in the twindow function can be both inclusive and exclusive.

The adverbs listed above can be used iteratively.

The operations of each adverb are conducted in left-to-right order. Take X <operator> :E:L Y for example, the leftmost adverb :E is first applied to each element in X and Y (X(i) and Y(i)), and then the next adverb :L is used to apply the operator for each X(i). The result is returned after all adverbs are applied.

a=1 2 3
b=4 5 6
c=(a,b)
re=c +:E:L c
re
// output
(1 2 3
- - -
2 3 4
3 4 5
4 5 6
,4  5  6
-- -- --
8  9  10
9  10 11
10 11 12
)

Rules to Dissemble and Assemble

In general, a vector is dissembled into scalars, a matrix into columns (vectors), a tuple into multiple elements (of different data forms), and a table into rows (represented by dictionaries).

In the assembly phase, the results of scalar type are merged to form a vector, vectors to a matrix, dictionaries to a table, and other data forms (which are inconsistent) into a tuple.

A template function iterates over a vector/tuple by elements, a matrix by columns, and a table by rows.