1.30.19

DolphinDB Server

New Features

  • Added new configuration parameters memLimitOfQueryResult and memLimitOfTaskGroupResult to restrict the memory usage of the intermediate and final results of queries; new function getQueryStatus to monitor the memory usage and execution status of the query.

  • Added new functions isPeak and isValley to determine if the current element is the peak/valley of the neighboring elements.

  • Added new function rowAt(X, Y). Return the element in each row of X based on the index specified by the corresponding element of Y.

  • Added new functions rowImin and rowImax to get the index of the extreme value in each row.

  • Added new machine learning function gmm to support Gaussian mixture model (GMM) clustering algorithms.

  • Added new function valueChanged to detect the change between elements by comparing the current element with adjacent elements.

  • Added new functions msum2 and tmsum2 to calculate the sum of squares in a sliding window.

  • Added new functions prevState and nextState to find the element with a different state before/after the current element. (Consecutive elements with the same value are considered to be of the same state.)

  • Added new function getSupportBundle. Return a file of support bundle containing system configuration and database information.

  • Added new functions topRange and lowRange. For each element in X, return the maximum length of a window to the left of X where it is the max/min. The functions are also supported in the reactive state engine (createReactiveStateEngine).

  • Added new parameter cumPositiveStreak for the reactive state engine (createReactiveStateEngine).

  • Added new streaming engine dual ownership reactive state engine (createDualOwnershipReactiveStateEngine) with support for parallel computing of data with 2 grouping methods and different metrics.

  • Introduced new table object "IPCInMemoryTable", interprocess in-memory table. Added related functions createIPCInMemoryTable, loadIPCInMemoryTable, dropIPCInMemoryTable and readIPCInMemoryTable. Interprocess in-memory table can be used in streaming scenarios to enable efficient data transfer between the DolphinDB server and client on the same machine.

  • Added new function stretch to stretch a vector evenly to the specified length.

  • Added new function getTransactionStatus to get the status of transactions. Added new command imtForceGCRedolog to skip the garbage collection of a transaction with the specified ID.

  • Added new module "ops" for database operations. This module contains some commonly-used scripts for operations such as cancelling unfinished jobs in the cluster, viewing disk usage of a DFS table, deleting recovering partitions, closing inactive sessions, etc.

  • Added new function setLogLevel to dynamically adjust the log level on the current node.

Improvements

  • getClusterPerf(true) returns the information on all controllers in a high-availability cluster. This function also adds a return value isLeader to indicate whether the controller is the leader of the raft group.

  • When using function restore, loadBackup, or getBackupMeta to access the backup partitions in a database whose chunk granularity is at TABLE level, the physical index is no longer required when specifying the parameter partition.

  • Function getRecoveryTaskStatus adds a new return value FailureReason to display the reason for the recovery task failure.

  • Optimized the compression algorithm for backup.

  • If a jobId does not exist when using cancelJob, the system no longer throws an exception. Instead, it outputs the error message with the jobId to the log.

  • Now can specify the configuration parameter persistenWorkerNum for a high-availability stream table.

  • Added new parameter forceTriggerTime to createSessionWindowEngine to trigger the calculation in the last window if useSystemTime=false.

  • When processing standard stream tables with streamFilter, you can now specify metacode of Boolean expressions for the filter condition.

  • You can include the time column and/or join column from the left or right table as the output column(s) in the the parameter metrics of functions createEqualJoinEngine, createAsofJoinEngine and createLookupJoinEngine.

  • The parameter keyPurgeFilter of createReactiveStateEngine must be metacode of Boolean expressions, otherwise an error will be raised.

  • The parameter metrics of createLookupJoinEngine can be a tuple.

  • Optimized the performance of select count(*) when the time granularity of a group by clause is more coarse-grained than that of a partition.

  • Optimized the performance of the following functions when calling function rolling: cumsum, cummax, cummin, cumprod, and mcount.

  • tar.gz file for offline server installation.

  • A subscription starts from the latest incoming data if the persisted offset cannot be found.

  • You can specify 00:00:00 for the parameter sessionEnd of function createDailyTimeSeriesEngine to indicate the end time is 00:00:00 of the next day (i.e., 24:00:00 of the day).

  • The number of rows in the result set of fj is limited to a maximum of 2 billion rows.

Issues Fixed

  • Garbage collection of redo log got stuck when data was continuously written to an OLAP database.

  • A node was wrongly considered alive by the controller after graceful shutdown.

  • Partition locks were prematurely released due to timeout before a transaction was resolved, which led to write failure.

  • Query conditions were wrongly processed when backing up data by specifying the conditions.

  • Server crashed when the machine load was excessively high.

  • When a cluster was restarted after a DFS table was updated, the original physical directories may not be recycled.

  • Serialization failure of symbol base caused read errors.

  • Streaming subscription failed to obtain data that was in memory but had been deleted from disk.

  • Server crashed when the number of columns inserted by appendForJoin did not match the table schema of the left or right table of a join engine.

  • When the parameter updateTime of function createSessionWindowEngine was specified and the output table was not a keyed table, the calculation could not be triggered after 2 * updateTime when a record arrived.

  • Server crashed when data was continuously ingested to a daily time series engine after the session end.

  • Failed to create a lookup join engine when the right table was specified as a shared keyed table.

  • If a node was restarted when a stream table was persisted to disk, data loss and decompression failure may occur.

  • Server crashed when specifying a time column (of a big array form) for dateColumn and timeColumn of replay and setting absoluteRate=false.

  • No error was reported when specifying a user-defined function with a constant return value for the metrics of a reactive state engine.

  • Server crashed when specifying temporary variables for metrics of createAnomalyDetectionEngine.

  • When using SQL update with context by, if the first column was set to integral type and the subsequent columns were set to floating-point types, values in the floating-point columns were rounded.

  • Concurrent pivot by queries may get stuck with few number of worker threads.

  • Server crashed when using HINT_EXPLAIN to query data from a three-level partitioned table.

  • Incorrect results when using function binsrch on a subarray of STRING type.

  • Function cast returned empty when converting a vector of STRING type to a tuple.

  • When aggregating multiple INT128 columns of an in-memory table, an error "The function min for reductive operations does not support data type INT128" occurred.

  • getFunctionView did not return some function view bodies.

  • Server crashed when an empty tuple was appended to itself and then loaded.

  • Server crashed when using interval interpolation in a SQL query, if the granularity of the data type specified for the time range in the where clause is greater than the time granularity specified by the duration parameter of interval.

  • Server crashed when using the function twindow in a SQL query.

  • update on a DFS table failed when the set column names did not match the original column names (including case sensitive inconsistencies).

  • When the function iterate was included in the metrics of a reactive state engine and the data cleaning mechanism was enabled, if data was inserted while the historical data was being cleaned, an error "vector::_M_default_append" was reported.

  • When calling matrix([[datehour(0)],[datehour(0)]]) to create a matrix, an error "The data object for matrix function can't be string or any type" was reported.

  • When specifying countNanInf for the parameter aggs of function wj, an error "An window join function must be an aggregate function" was reported.

  • If no group was specified for createDailyTimeSeriesEngine, any uncalculated data from the previous day would be merged into the first window of the following day.

  • The first window's calculation result across days in a daily time series engine was incorrect. Additionally, when the function fill was used to fill in NULL values, data outside the session was output.

  • Tasks in In-Progress state could not be recovered during online recovery.

  • Server crashed when specifying useSystemTime=true and mode for metrics of createTimeSeriesEngine.

  • When specifying the tmove or move function for the metrics of createReactiveStateEngine, the server would crash if X was of the STRING or SYMBOL type.

  • Failed to insert data with tableInsert when splitting and assigning a stream table with streamFilter.

  • The insert failure of streamFilter may cause session disconnection due to excessively long error messages.

  • When specifying function firstNot or lastNot for the metrics of createTimeSeriesEngine or createReactiveStateEngine and setting fill=`ffill, the output did not match expectations.

  • Server crashed when specifying function mfirst or mlast for the metrics of createReactiveStateEngine, and X was of FLOAT, SHORT, CHAR, BOOL, INT128, STRING, or SYMBOL type.

  • Executing function tableInsert changed the atomic level of a database from 'CHUNK' to 'TRANS'.

  • When specifying tm-functions for the metrics of createReactiveStateEngine and the window parameter of the function is set to y, M or B, the calculation result was incorrect.

  • Inconsistent STRING columns between replicas after online recovery.

  • When passing TIME type data to index of resample and setting origin="end" and rule="D", an error "Invalid value for HourOfDay (valid values 0 - 23): 39" was reported.

  • An error was reported when administrators (except the super admin) granted/denied/revoked permissions to themselves.

  • Calculating imin or imax with byRow on a matrix with an empty row returned incorrect results.

  • Function getControllerPef returned incorrect agent site when a controller crashed.

  • When dataSync was not configured, an error occurred when dynamically calling the addNode function to add a node.