Config#

class swordfish._config.Config#

A class that manages various configuration settings.

config_dict: Dict[str, ConfigDescriptor]#
dataSync: int#

Whether database logs are forced to persist to disk before the transaction is committed.

  • If dataSync=1 (default), the redo logs, data, and metadata are forced to persist to disk.

  • If dataSync=0, the operating system will decide when to write the log files to disk.

decimalRoundingMode: str#

Specifies the rounding mode for Decimal type. The default is "trunc", meaning the decimal part is truncated. Set it to "round" to round the decimal part. It is applicable to the following conversion scenarios:

  • Parsing floating-point numeric strings to DECIMAL type (e.g., loading files with loadText).

  • Converting floating-point numbers to Decimal type.

  • Converting Decimal values to Integral type.

  • Converting high-precision Decimal values to low-precision Decimal values.

Note

The rounding mode for the first case is rounding. For other scenarios, truncation was used.

enableChunkGranularityConfig: bool#

Specifies the chunk granularity to determine the level of the lock of a transaction. When writing to a chunk, the transaction locks it to prevent other transactions from writing to it.

The default value is True. You can specify the chunk granularity with the parameter chunkGranularity of function database.

If set to False, the chunk granularity is at the table level, i.e., each tablet of a partition is a chunk. Concurrent writes to different tables in the same partition are thus allowed.

enableConcurrentDimensionalTableWrite: bool#

Whether to allow conducting concurrent write or update/delete on dimension tables.

The default value is False, indicating concurrent write and update/delete is disabled for dimension tables.

enableDFS: bool#
enableInsertStatementForDFSTable: bool#

Whether to enable the insert into statement for DFS tables. The default value is True.

enableLocalDatabase: bool#

Whether to enable the creation of a local database. The default value is False.

home: Path#

The DolphinDB home directory (<HomeDir>). It can only be specified in command line.

ignoreSpecialCharacterInPartitionId: bool#

Whether to ignore ":" and "." in partitioning columns when creating partition directories for value-partitioned tables with partitioning columns of STRING or SYMBOL type.

The default value is False. The “.” and “:” will be included as part of the path, which means data written to the “.a:bc.” and “abc” partitions will be separated into different paths.

If set to True, keys like ".a:bc." and “abc” of the partitioning column will map to the same directory path since “:” and “.” are ignored. So data written to partitions “.a:bc.” and “abc” will both end up being stored under “abc”.

init: Path#

This file is executed when the system starts. The default file is {package install dir}/asset/dolphindb.dos.

It usually contains definitions of system-level functions that are visible to all users and cannot be overwritten.

license: Path#
localSite: str#

The LAN information of the node, including host address, port number and alias of the node, separated by ":".

logFile: Path#

The path and name of the log file. It displays the Swordfish configuration specifications, warnings, and error messages.

logicOrIgnoreNull: bool#

Whether function or ignores null values.

When set to False (default), it always returns NULL regardless of the value of the other operand, which matches the behavior in prior versions.

When set to True:

  • If the other operand is non-zero, it returns True.

  • If the other operand is zero, it returns False.

  • If the other operand is a null value, it returns NULL.

logLevel: str#

The level equal to and above which logs are printed to help users locate the error message in the log. From the lowest to the highest level, the possible values are DEBUG, INFO, WARNING, and ERROR. The default value is ERROR.

logRetentionTime: float#

To prevent excessive resource usage from old logs, the logRetentionTime parameter can be used to specify the amount of time (in days) to keep a log file before deleting it.

The default value is 30 (of floating-point data type). For example, 0.5 means 12 hours. If set to 0, log files are not deleted.

marketHolidayDir: Path#

The directory to the file of market holidays. Either an absolute or relative directory can be specified.

The system searches the relative directory in the following order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable.

The default directory is {package install dir}/asset/marketHoliday. The file must be a single-column CSV file with Date data. Based on the CSV files under marketHolidayDir, a trading calendar is created with the file name as its identifier.

Notes

  • Weekends are recognized as holidays by default; therefore, only weekday holidays need to be filled in the file.

  • It is recommended to name the file in Market Identifier Code, e.g., "XNYS.csv".

maxBlockSizeForReservedMemory: float#

The maximum size (in units of KB) of the memory block that DolphinDB allocates for each memory request when its available memory is less than reservedMemSize. The default value is 64.

It is not recommended to set it too high as exceptions or crashes may occur if there isn’t enough memory left for critical database operations.

maxConnections: int#

The maximum number of connections to the local node.

maxLogSize: int#

The system will archive the Swordfish log after it reaches the specified size limit (in MB). The default value is 1024 and the minimum value is 100.

Upon reaching this limit, Swordfish automatically generates a new log with a timestamp prefix in seconds. The prefix is in the format of <date><seq>, e.g., 20181109000. <seq> has 3 digits and starts with 000.

maxMemSize: float#

The maximum memory (in units of GB) allocated to Swordfish. The default value is 0, meaning no limits on memory usage. The value of maxMemSize cannot exceed the machine memory.

maxPartitionNumPerQuery: int#

The maximum number of partitions that a single query can search. The default value is 65536.

maxPubConnections: int#

The maximum number of subscriber nodes that the publisher node can connect to. The default value is 64.

For the node to serve as a publisher, we must set maxPubConnections > 0.

maxSubConnections: int#

The maximum number of publisher nodes that the subscriber node can connect to. The default value is 64.

maxSubQueueDepth: int#

The maximum depth (number of records) of a message queue on the subscriber node. The default value is 10,000,000.

memLimitOfAllTempResults: float#

The total memory limit (in GB) for all temporary results generated during SQL distributed queries. The default value is 20% * maxMemSize.

If the memory usage exceeds the limit, temporary results are spilled to disk.

memLimitOfQueryResult: float#

The memory limit for a query result. The default value is min(50% * maxMemSize, 8G), and it must be smaller than 80% * maxMemSize.

memLimitOfTaskGroupResult: float#

In the Map phase of MapReduce, a single query is divided into several tasks, among which the remote tasks are sent to remote nodes in batches (task groups). memLimitOfTaskGroupResult is used to set the memory limit of a task group sent from the current node.

The default value is min(20% * maxMemSize, 2G), and it must be smaller than 50% * maxMemSize.

moduleDir: Path#

The directory for the module files. It can be an absolute path or a relative directory, with modules as the default.

The system searches the relative directory modules in the following order: home directory, the working directory, and the directory with the swordfish package.

newValuePartitionPolicy: str#

How new data outside of existing VALUE partitions are handled in databases. It can be “add”, “skip”, or “fail”.

  • "add" (default): Creates new VALUE partitions to add all data.

  • "skip": Appends data that belongs to existing VALUE partitions and ignores out-of-scope data.

  • "fail": Throws exception if out-of-scope data is included. None of the data will be written. In most cases, "add" is recommended.

nullAsMinValueForComparison: bool#

Whether a null value is treated as the minimum value in data comparison. The default value is True.

If it is set to False, the result of comparison involving null values is NULL.

OLAPCacheEngineSize: float#

The capacity of cache engine in units of GB. After cache engine is enabled, data is not written to disk until data in cache exceeds 30% of OLAPCacheEngineSize.

The default value is 0.5. To enable the cache engine, we must set OLAPCacheEngineSize > 0 and dataSync = 1.

parseDecimalAsFloatingNumber: bool#

Whether decimal numbers are parsed to Double or Decimal64 type.

  • If set to True (default), decimals are parsed to Double floating point numbers.

  • If set to False, decimals are parsed to Decimal64 fixed point numbers.

persistenceDir: Path#

The directory where shared stream tables are persisted to. To enable persistence, persistenceDir must be specified.

regularArrayMemoryLimit: float#

The limit on the maximum memory size (in MB) of a regular array. Its value must be a power of 2. The default value is 2048 (MB).

The actual available memory is min(regularArrayMemoryLimit, maxMemSize/2). If the memory occupied by an array exceeds this limit, the system will create a big array instead.

removeSpecialCharInColumnName: bool#

Whether to normalize column names.

  • If set to False (default), column names can contain special characters (except underscores) or start without letters.

  • If set to True, column names are normalized.

reservedMemSize: float#

A positive number specified in units of GB. When the available memory in Swordfish is less than reservedMemSize, Swordfish only allocates a memory block of limited size (as specified by maxBlockSizeForReservedMemory) for each memory request.

reservedMemSize is provided to restrict the memory allocation for each memory request to improve the likelihood that there is enough memory for critical operations that use a small amount of memory (error reporting, rollbacks, etc.).

For example, when data writes fail due to insufficient memory, the transactions can be rolled back to guarantee data consistency. If the parameter is not specified, the system sets reservedMemSize = 5% * maxMemSize and reservedMemSize must be between 64MB and 1GB.

stdoutLog: int#

Where to output the system log. It can be:

  • 0: dolphindb.log;

  • 1 (default): stdout;

  • 2: both stdout and dolphindb.log.

subExecutors: int#

The number of message processing threads in the subscriber node. Only when subscription is enabled is this parameter relevant.

The default value is 8. If it is set to 0, it means the thread can conduct message parsing and can also process messages.

subThrottle: int#

A non-negative integer in milliseconds, indicating the interval at which the system checks whether the throttle parameter in the subscribe function has been reached.The default value is 1000.

If the interval specified by the throttle parameter in subscribe is less than subThrottle, the handler parameter in subscribe will be triggered to process messages at an interval of subThrottle.

To set throttle to less than 1 second, you need to modify the configuration parameter subThrottle first. For example, to set throttle = 0.001 (second), please set subThrottle = 1 first.

Notes

This parameter is only valid if the parameter batch_size is specified in function subscribe.

tcpNoDelay: bool#

Whether to enable the TCP_NODELAY socket option. The default value is False.

tcpUserTimeout: int#

Set the socket option TCP_USER_TIMEOUT. The default value is 300000 (in ms).

tempResultsSpillDir: Path#

The directory for storing temporary results generated during distributed queries when they exceed memory limits. The default directory is <HomeDir>/tempResults.

Temporary results exceeding the memory limit (specified by memLimitOfAllTempResults) are spilled to the directory. Files are automatically deleted after query completion.

Notes

This directory is recreated after Swordfish initialization, deleting any existing contents.

TSDBCacheEngineSize: float#

A positive number indicating the capacity (in GB) of the TSDB cache engine. The default value is 1.

The memory used by the cache engine may reach twice the set value as an extra memoryblock will be allocated to cache the incoming data while data in the original memory is being flushed to disk.

  • If the flush process is not fast enough, the newly allocated memory may also reach TSDBCacheEngineSize and thus block the writer thread.

  • If the parameter is set too small, data in the cache engine may be flushed to disk too frequently, thus affecting the write performance;

  • If set too high, a large volume of cached data is not flushed to disk until it reaches TSDBCacheEngineSize (or after 10 minutes).

  • If power failure or shutdown occurs in such cases, numerous redo logs are to be replayed when the system is restarting, causing a slow startup.

tzdb: Path#

The directory of the time zone database. The default value is {package install dir}/asset/tzdb.

warningMemSize: float#

When memory usage exceeds warningMemSize (in units of GB), the system will automatically clean up the cache of some databases to avoid OOM exceptions. The default value is 20.

workerNum: int#

The size of worker pool for regular interactive jobs. The default value is the number of CPU cores.

build()#

Builds a list of command-line arguments based on the configuration.

Returns:

A list of command-line arguments.

Return type:

list

read_config_from_toml(file_path)#
Parameters:

file_path (Path)