1.30.22

DolphinDB Server

New Features

  • Added new function appendTuple! to append a tuple to another. (1.30.22.4)

  • Added new configuration parameter appendTupleAsAWhole to specify whether the tuple should be appended as an embedded tuple element, or if each of its elements should be appended independently to the target tuple. (1.30.22.4)

  • Added login information in logs, including login user, IP, port, status, etc. (1.30.22.4)

  • Added privilege VIEW_OWNER to support a user/group to create function views using addFunctionView. (1.30.22.4)

  • Support for partition pruning when the partitioning column is of the NANOTIMESTAMP type. (1.30.22.4)

  • Added new parameter isSequential to the plugin.txt to mark a function as order-sensitive or not. (1.30.22.4)

  • Added the cumdenseRank function to perform dense ranking of elements within cumulative windows. (1.30.22.3)

  • Added a new "dataInterval" option to the triggeringPattern parameter of the createCrossSectionalEngine function. This option enables calculations to be triggered based on timestamps from the input data. (1.30.22.3)

  • Added function parseJsonTable to parse a JSON object to an in-memory table. (1.30.22.2)

  • Added new configuration parameter tcpUserTimeout to set the socket option TCP_USER_TIMEOUT. (1.30.22.2)

  • Removed function getClusterReplicationMetrics. Added function getSlaveReplicationQueueStatus as an inheritance of getClusterReplicationMetrics.getSlaveReplicationQueueStatus retrieves the status of each execution queue in the slave clusters. (1.30.22.2)

  • Added configuration parameter clusterReplicationQueue to set the number of execution queues on each controller of the slave clusters. (1.30.22.2)

  • Added configuration parameter clusterReplicationWorkerNum to set the number of workers on each data node of the slave clusters. (1.30.22.2)

  • Added configuration parameter enableCoreDump to enable core dumps. It is only supported on Linux. (1.30.22)

  • Added configuration parameter disableCoreDumpOnShutdown to specify whether to generate core dumps on a graceful shutdown. It is only supported on Linux. (1.30.22)

  • Added configuration parameter allowMissingPartitions to specify the behavior when incoming data contains new partition values that do not match any existing partitions. (1.30.22)

  • Added function listRemotePlugins to obtain a list of available plugins. Added function installPlugin to download a plugin. (1.30.22)

  • Added configuration parameter volumeUsageThreshold to set the upper limit of the disk usage of a data node. (1.30.22)

  • Added function writeLogLevel to write logs of the specified level to the log file. (1.30.22)

  • Added function sessionWindow to group time-series data based on the session intervals. (1.30.22)

  • Added function summary to generate summary statistics of input data, including min, max, count, avg, std, and percentiles. (1.30.22)

  • Added functions encodeShortGenomeSeq and decodeShortGenomeSeq to encode and decode DNA sequences. (1.30.22)

  • Added function genShortGenomeSeq to perform DNA sequences encoding within a sliding window. (1.30.22)

  • Added function gramSchmidt to implement the Gram–Schmidt orthonormalization. (1.30.22)

  • Added function lassoBasic that has equivalent function to lasso but takes vectors as input arguments. (1.30.22)

  • Added 26 TopN functions: (1.30.22)

    • m-functions: mskewTopN, mkurtosisTopN
    • cum-functions: cumsumTopN, cumavgTopN, cumstdTopN, cumstdpTopN, cumvarTopN, cumvarpTopN, cumbetaTopN, cumcorrTopN, cumcovarTopN, cumwsumTopN, cumskewTopN, cumkurtosisTopN
    • tm-functions: tmsumTopN, tmavgTopN, tmstdTopN, tmstdpTopN, tmvarTopN, tmvarpTopN, tmbetaTopN, tmcorrTopN, tmcovarTopN, tmwsumTopN, tmskewTopN, tmkurtosisTopN

  • Added function initcap to set the first letter of each word in a string to uppercase and the rest to lowercase. (1.30.22)

  • Added functions splrep and splev for cubic spline interpolation. (1.30.22)

  • Added function scs to compute the optimal solution of linearly constrained linear or quadratic programming functions. (1.30.22)

  • Added function temporalSeq to generate time series at specified intervals. (1.30.22)

  • Added functions base64Encode and base64Decode to encode and decode Base64 digits. (1.30.22)

  • Added function addFunctionTypeInferenceRule to specify the inference rule of user-defined functions in DolphinDB JIT version. (1.30.22)

  • Added support for COMPLEX data type in DolphinDB JIT version. (1.30.22)

  • Added function createStreamDispatchEngine to create a streaming data dispatch engine. (1.30.22)

Improvements

  • Optimized the performance of function dropTable when deleting a partitioned table with over 100,000 partitions. (1.30.22.4)

  • The divisor of div/mod now can be negative numbers. (1.30.22.4)

  • Optimized transactions on compute nodes. (1.30.22.2)

  • Added parameter keepRootDir to function rmdir to specify whether to keep the root directory when deleting files. (1.30.22.2)

  • The license function obtains license information from memory by default. (1.30.22.2)

  • An empty table can be backed up by copying files. (1.30.22.2)

  • Optimized asynchronous replication (1.30.22.2):

    • After asynchronous replication is enabled globally, the system now allows operations on slave cluster databases which are not included in the replication scope.
    • The mechanism for pulling replication tasks from the master to the slave clusters has been improved.

  • <DataNodeNotAvail> error message now provides more details. (1.30.22.2)

  • A user-defined function allows the default value of a parameter to be an empty tuple (represented as []). (1.30.22.1)

  • Added user access control to the loadText function. (1.30.22.1)

  • Modifications made to user access privileges are logged. (1.30.22.1)

  • The resample function can take a matrix with non-strictly increasing row labels as an input argument. (1.30.22.1)

  • Optimized the join behavior for tuples. (1.30.22.1)

  • A ternary function can be passed as an input argument to the template accumulate in a reactive state engine. (1.30.22.1)

  • Added parameter validation to streamEngineParser: If triggeringPattern='keyCount', then keepOrder must be true. (1.30.22.1)

  • Configuration parameters localExecutors and maxDynamicLocalExecutor were discarded. (1.30.22)

  • Functions window and percentChange can be used as state functions in the reactive state engine. (1.30.22)

  • Support JOIN on multiple partitioned tables. (1.30.22)

  • Optimized the performance when using the dropTable function to delete a table with a large number of partitions. (1.30.22)

  • Support SQL keywords in all uppercase or lowercase. (1.30.22)

  • Support comma (,) to CROSS JOIN tables. (1.30.22)

  • Support line breaks for SQL statements, while keywords with multiple words, such as ORDER BY, GROUP BY, UNION ALL, INNER JOIN, cannot be split into two lines. (1.30.22)

  • The implementation of select * from a join b is changed from select * from join(a, b) to select * from cj(a, b). (1.30.22)

  • Support operator <> in SQL statements, which is equivalent to !=. (1.30.22)

  • Support keyword NOT LIKE in SQL statements. (1.30.22)

  • When LEFT JOIN, LEFT SEMI JOIN, FULL JOIN or EQUI JOIN on columns containing NULL values: (1.30.22)

    • In the previous versions: a NULL value is matched to another NULL.
    • Since the current version: a NULL value cannot be matched to another NULL.

  • For function sqlDS, a DFS table partitioned by DATEHOUR selected in sqlObj will now be correctly filtered by date. (1.30.22)

  • For the "Status" column returned by function getRecoveryTaskStatus, the previous status "Finish" is now changed to "Finished", "Abort" to "Aborted". (1.30.22)

  • Added inplace optimization fields, i.e., inplaceOptimization and optimizedColumns, when using HINT_EXPLAIN to check the execution plan of a GROUP BY clause when algo is "sort". (1.30.22)

  • The column name specified with the rename!, replaceColumn!, dropColumns! functions are no longer case sensitive. (1.30.22)

  • Added new parameters swColName and checkInput for the lasso and elasticNet functions to specify the sample weight and validation check, respectively. Added new parameters swColName for the ridge function. (1.30.22)

  • Added parameters x0, c, eps, and alpha for function qclp to specify absolute value constraints, solving accuracy, and relaxation parameters. (1.30.22)

  • Functions loadText, pLoadText, and extractTextSchema now can load a data file that contains a record with multiple newlines. (1.30.22)

  • The delimiter parameter of the loadText, pLoadText, loadTextEx, textChunkDS, extractTextSchema functions can be specified as one or more characters. (1.30.22)

  • When importing a table using function loadTextEx, an error will be reported if the table schema does not match the schema of the target database. (1.30.22)

  • Added check for the schema parameter of function loadTextEx. Since this version, the table specified by schema MUST NOT be empty, and the "name" and "type" columns must be of STRING type. (1.30.22)

  • Added new parameter tiesMethod, which is used to process the group of records with the same value, for the following moving TopN functions: mstdTopN, mstdpTopN, mvarTopN, mvarpTopN, msumTopN, mavgTopN, mwsumTopN, mbetaTopN, mcorrTopN, mcovarTopN. (1.30.22)

  • Optimized the prediction performance of function knn. (1.30.22)

  • The time series engine and daily time series engine now can output columns holding array vectors. (1.30.22)

  • Optimized the performance of the moving function used in the reactive state engine. (1.30.22)

  • The anomaly detection engine now can specify multiple grouping columns for parameter keyColumn. (1.30.22)

  • Added new parameter sortByTime for the createWindowJoinEngine and createAsOfJoinEngine functions to determine whether the result is returned in the order of timestamps globally. (1.30.22)

  • The streaming engine can now be shared with the share function/statement for concurrent writes. (1.30.22)

  • An error will be reported when using the share function/statement or the enableTableShareAndPersistence function to share the same table multiple times. (1.30.22)

  • An error will be reported if the data of INT type is appended to a SYMBOL column of the left table of a window join engine. (1.30.22)

  • DolphinDB JIT version supports the join operator (<-). (1.30.22)

  • The isort function in JIT version can take a tuple with vectors of equal length as input. (1.30.22)

  • The if expression in JIT version supports the in operator. (1.30.22)

  • Vectors can be accessed with Boolean index in JIT version. (1.30.22)

  • Support comments with multiple /**/ sections in one line. (1.30.22)

  • The function stringFormat now supports: data type matching, format alignment, decimal digits, and base conversion. (1.30.22)

  • The second parameter of function concat can be NULL. (1.30.22)

  • Function take can take a tuple or table as input. (1.30.22)

  • Function stretch can take a matrix or table as input. (1.30.22)

  • Functions in and find support table with one column. (1.30.22)

  • When the parameter moduleDir is configured as a relative path, the system searches the modules under the homeDir/modules directory. (1.30.22)

  • The result of function in, binsrch, find, or asof takes the same format as the input argument Y. (1.30.22)

  • An error is raised when passing a tuple to function rank. (1.30.22)

Issues Fixed

  • Data contention when updating a table schema led to OOM problem and server crash. (1.30.22.4)

  • The backup might get stuck when the backup directory (backupDir) is on NFS. (1.30.22.4)

  • The memory access out of bounds error occured when attempting to close a connection that was created after setting the maximum number of connections using setMaxConnections. (1.30.22.4)

  • When joining partitioned tables using a statement that did not conform to SQL standards, referencing a column from the left table in the where clasue caused the server to crash. (1.30.22.4)

  • If creating an IPC in-memory table failed, creating another one with the same name caused the server to crash. (1.30.22.4)

  • An error was reported when the filtering condition in a distributed query contained a comparison between operands of SECOND and INT type. (1.30.22.4)

  • The SYMBOL type in an IPC in-memory table was not compatible with the STRING type. (1.30.22.4)

  • An error message "getSubChunks failed, path'/xx' does not exist" was reported when restoring data to a newly-created database. (1.30.22.2)

  • The elements accessed based on labels by loc function were incorrect. This issue was introduced in version 1.30.22. (1.30.22.2)

  • Scale loss occurred when restoring DECIMAL data. (1.30.22.2)

  • If the parameter atomic of function database was set to 'CHUNK', the versions of metadata on the controller and data nodes may be inconsistent if a transaction involved multiple chunks. (1.30.22.2)

  • Passing a non-string variable to the parameter label of function interval crashed the server. (1.30.22.2)

  • For a table partitioned by temporal values, queries with where conditions on the partitioning column were slow. This issue was introduced in version 1.30.22. (1.30.22.2)

  • The overflowed intermediate result of function mprod caused server crash. (1.30.22.2)

  • Concurrent execution of restore (and other) transactions may result in inconsistent metadata after server restart. (1.30.22.2)

  • On Windows, the files function returned inaccurate fileSize values for files exceeding 2 GB. (1.30.22.1)

  • In a high-availability cluster, if an error occurred during serialization when using addFunctionView, the function was not cleared from memory. (1.30.22.1)

  • In a high-availability cluster, adding a function view containing plugin methods to a controller caused failures in other controllers. (1.30.22.1)

  • Users with DB_MANAGE privilege failed to grant permissions to other users. (1.30.22.1)

  • Adding a node may cause backup errors. (1.30.22.1)

  • Queries on DFS tables using COMPO partitioning may cause data loss if the query: (1.30.22.1)

    • Did not use aggregate functions, order-sensitive functions, row reduce functions (such as rowSum), or fill functions (such as ffill) in the select statement.

    • Used one of the partitioning columns (except the last one for COMPO partitioning) as a pivot-by column.

  • If an error occurred in a symbol base file, reloading the file caused server crash. (1.30.22.1)

  • Specifying a tuple containing functions or expressions with multiple returns for the metrics parameter of createReactiveStateEngine caused the server to crash. (1.30.22.1)

  • [D20-11604] Fixed unexpected results returned by functions mstd, mstdp, mvar, and mvarp when processing consecutive identical numbers (non-DECIMAL) within a window due to floating-point precision issues. A precision check has now been added and the return value is 0 in such cases. (1.30.22)

  • When querying a large DFS table using the SQL keyword TOP or GROUP BY, an error was potentially raised. (1.30.22)

  • When a SQL query specified a column name that couldn't be recognized, the error message returned contained an incorrect column name instead of the actual unrecognized column name from the query. (1.30.22)

  • Failures to write to a partition of a DFS table with many columns could cause the server to crash. (1.30.22)

  • Concurrently loading and deleting multiple tables in a database could cause subsequent loadTable operations to fail with an error reporting it cannot find the .tbl file. (1.30.22)

  • The head and tail functions could not be used in aggregate functions. This bug was introduced in DolphinDB 1.30.18. (1.30.22)

  • A deadlock could occur when concurrently renaming a dimension table via renameTable and querying the same table. (1.30.22)

  • When querying a table with a large number of partitions using a SQL query with BETWEEN...AND... for partition pruning, the error The number of partitions [xxxxx] relevant to the query is too large could be raised. (1.30.22)

  • Using calculations or functions in a CASE WHEN condition could crash the server. (1.30.22)

  • Using the DISTINCT keyword in SQL queries could return incorrect results. (1.30.22)

  • When querying a VALUE or RANGE partitioned DFS table, if the SELECT clause and GROUP BY clause both applied the same time conversion function (e.g. date()) to the partitioning column, but used different aliases for that column, incorrect results could be returned. (1.30.22)

  • When deleting data from a partitioned table using a SQL DELETE statement, if all nodes storing the replicas for the relevant partition were offline, the error chunktype mismatched for path could be raised. (1.30.22)

  • The use of local executors could lead to deadlock situations during task scheduling. (1.30.22)

  • In the DolphinDB JIT version, when appending large amounts of data to a reactive state engine (createReactiveStateEngine) that used user-defined functions, incorrect results could be returned. (1.30.22)

  • A deadlock may occur when unsubscribeTable was called from multiple nodes simultaneously. (1.30.22)

  • Server crashed when the capitalization of the column names specified in metrics and input tables of a left semi join engine (createLeftSemiJoinEngine) was inconsistent. (1.30.22)

  • Server crashed when appending data to a stream table and persisting the table at the same time. (1.30.22)

  • After DROP table was called to delete a stream table, the table could not be deleted or unsubscribed from. (1.30.22)

  • Syntax parsing issues: statements such as "/" == "a" could not be parsed correctly. (1.30.22)

  • An additional column was output when the second parameter of function ols consisted solely of 0. (1.30.22)

  • Server crashed due to parsing failure when the parameter aggs of function wj was not compliant. (1.30.22)

  • The result of function expr was incorrect if a DATEHOUR argument was passed. (1.30.22)

  • The web interface could not be accessed properly if the parameter webLoginRequired was configured to true. (1.30.22)

  • Incorrect results were returned when using cast to convert SYMBOL data. (1.30.22)

  • Function nullFill failed to fill the NULL values returned by function bucket. (1.30.22)

  • When a user-defined anonymous aggregate function was called with twindow in another user-defined function, an error func must be an aggregate function. was raised. (1.30.22)

  • When a DolphinDB process was started, server crashed if a script (as configured with parameter run) containing function submitJob was executed. (1.30.22)