Higher-Order Functions

Introduction

A higher order function can extend or enhance a function or an operator. A higher-order function takes a function and some objects as input. It works like a pipe between the function and its input data. In general, the input data is dissembled into multiple pieces (which may or may not overlap) in a preset way at first; then the individual pieces of data are applied to the given function to produce results one at a time; finally all individual results are assembled into one object to return. The input data for a higher-order function can be vectors, matrices or tables, with the occasional use of scalars and dictionaries. With a higher-order function, complicated analytical tasks can be accomplished very efficiently with only a few lines of statements.

Syntax

Higher-order functions are always used together with operators, user-defined functions, or system functions. Higher-order function symbols start with the symbol colon ":" followed by an upper-case single character.

higherOrderFunctionName (<functionName>, functionParameter1, ...functionParameterN)

<functionName> :<higher order symbol> functionParameter

functionParameter1 <functionName> :<higher order symbol> functionParameter2

In DolphinDB, the first parameter of all higher-order functions must be a function name. To call more than one function in a higher-order function, you can specify the call higher-order function as the first parameter <functionName>, and then pass in the functions you want to call as argument to call. Please refer to the section about the higher-order function call.

Starting with version 1.30.17, higher-order functions always treat the first parameter as a function.

Higher-order Function Summary

The summary table below shows higher-order function names and applications.


Symbol	Name	Applications	Examples
:E	each	unary operators, binary operators, function calls
	peach	parallel version of `each`
:R	eachRight	binary operators
:L	eachLeft	binary operators
:P	eachPre	binary operators, function calls
:O	eachPost	binary operators, function calls
	pivot	function calls	Transpose rows and columns of raw data or grouped aggregation results
:A	accumulate	binary operators, function calls
:T	reduce	binary operators, function calls
:G	groupby	binary operators, function calls
:C	cross	binary operators, function calls
	pcross	parallel version of `cross`	Work with pivoting
	moving	binary operators, function calls (aggregate function)
	window	function calls	Similar to `moving`. Compared with the moving function, the window function has a more flexible window. The left/right boundary of the window specified in the `window` function can be both inclusive and exclusive.
	nullCompare	binary operators, function calls	Preserve null values in the calculation to keep consistent with the behavior of Python.
	loop	unary operators, binary operators, function calls, mixed return types	Import text files
	ploop	parallel version of `loop`
	all	binary operators, function calls
	any	binary operators, function calls
	call	function calls	Usually used with function `each` to simultaneously call a batch of functions
	pcall	parallel version of `call`
	unifiedCall	function calls	Similar to `call`. The size of args in `unifiedCall` is always 1.
:X	contextby	binary operators, function calls	Perform specified calculations in groups
	segmentby	binary operators, function calls
	rolling	binary operators, function calls	Calculate the beta value of APPL on the market (SPY)
	withNullFill	binary operators	Replace the null value with a specific value in the calculation
:H	byRow	function calls	Apply the function to each row of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple.
:V	byColumn	function calls	Apply the function to each column of a two-dimensional data object such as matrix, table, tuple, array vector, and columnar tuple.
	talib	function calls	To process data with DolphinDB functions in the same way as Python TA-lib
	tmoving	function calls	Apply the function/operator to a time-based sliding window (with the right boundary inclusive) of the given objects.
	twindow	function calls	Similar to `tmoving`. Compared with the tmoving function, `twindow` has more flexible windows. The left/right boundary of the window specified in the `twindow` function can be both inclusive and exclusive.

The adverbs listed above can be used iteratively.

The operations of each adverb are conducted in left-to-right order. Take X <operator> :E:L Y for example, the leftmost adverb :E is first applied to each element in X and Y (X(i) and Y(i)), and then the next adverb :L is used to apply the operator for each X(i). The result is returned after all adverbs are applied.

a=1 2 3
b=4 5 6
c=(a,b)
re=c +:E:L c
re

/* output
(1 2 3
- - -
2 3 4
3 4 5
4 5 6
,4  5  6
-- -- --
8  9  10
9  10 11
10 11 12
)
*/

Rules to Dissemble and Assemble

In general, a vector is dissembled into scalars, a matrix into columns (vectors), a tuple into multiple elements (of different data forms), and a table into rows (represented by dictionaries).

In the assembly phase, you can specify how to combine outputs from sub-tasks into the final output. Under the default DolphinDB rule, the results of scalar type are merged to form a vector, vectors to a matrix, matrices to a tuple, dictionaries to a table, and other data forms (which are inconsistent) into a tuple. Additionally, three other rules are available:

The Tuple rule always outputs a tuple
The Consistent rule assumes all sub results match the first sub result's type and form, using it to determine the final output's type and form.
The kdb+ rule outputs a tuple if any sub result is a vector.

Here's a simple example using the :E function pattern (see each):

vec = 1..9
tp = (vec, vec, vec)
typestr copy:EC(tp) // FAST INT MATRIX
typestr copy:ED(tp) // FAST INT MATRIX
typestr copy:EU(tp) // ANY VECTOR
typestr copy:EK(tp) // ANY VECTOR

In this example, all :E sub-tasks return vectors of the same length and type. The D (DolphinDB) and C (Consistent) rules combine these sub results into a matrix. The U (Tuple) rule always returns a tuple regardless of input. And since the sub results contain vectors, the K (kdb+) rule also produces a tuple as the final output.

A higher-order function iterates over a vector/tuple by elements, a matrix by columns, and a table by rows.