Serialization

DolphinDB Python API 3.0.2.1 adds the io module with serialization and deserialization functions.

dump

Serialize an object and write it to an opened file object.

def dump(obj, file, *, types=None)

obj: The object to be serialized.
file: The file object to write to.
types: optional. Specifies the data types for serialization using type constants (defined in the dolphindb.settings module). The format depends on the type of obj:
- For dictionaries/dict-like objects: [int, int]. Two-element list where first element is key type and second element is value type, e.g., [DT_STRING, DT_INT])
- For DataFrames/table-like objects: dict. Dictionary with column names as keys and their data types as values, e.g., {'col1': DT_STRING, 'col2': DT_INT}
- For other types, use either:
  - int. Single integer representing the data type, e.g., DT_INT or DT_DECIMAL32
  - [int, int]. Two-element list. First element is the data type, second element is an additional parameter (usually scale for Decimal types). For example, [DT_DECIMAL32, 2]

Return value

None

load

Read and deserialize an object from an opened file object.

def load(file) -> Any

file: The file object to read from.

Return value

The deserialized object.

dumps

Serialize an object to a bytes object.

def dumps(obj, *, types=None) -> bytes

obj: The object to be serialized.
types: optional. Specifies the data types for serialization using type constants (defined in the dolphindb.settings module). The format depends on the type of obj:
- basic data types objects:
  - int. Single integer representing the data type, e.g., DT_INT or DT_DECIMAL32
  - [int, int]. Two-element list. First element is the data type, second element is an additional parameter (usually scale for Decimal types). For example, [DT_DECIMAL32, 2]
- dictionaries/dict-like objects: [int, int]. Two-element list where first element is key type and second element is value type, e.g., [DT_STRING, DT_INT])
- DataFrames/table-like objects: dict. Dictionary with column names as keys and their data types as values, e.g., {'col1': DT_STRING, 'col2': DT_INT}

Return value

bytes. The serialized object.

loads

Deserialize an object from a bytes object.

def loads(data) -> Any

data: A binary string containing the serialized data.

Return value

The deserialized object.

Examples

First import the dolphindb.io module:

from dolphindb import io

Example 1

Serialize to file and read back:

with open("data.bin", "wb") as f:
     io.dump(1, f)

with open("data.bin", "rb") as f:
     re = io.load(f)
     print(re)

# output: 1

Serialize to bytes object and read back:

data = io.dumps(1)
print(data)

# output: b'\x05\x00\x01\x00\x00\x00\x00\x00\x00\x00'

re = io.loads(data)
print(re)
# output: 1

Example 2

Serialize Decimal data. The types parameter only specifies the object type, not scale:

from decimal import Decimal
import dolphindb.settings as keys
data = io.dumps(Decimal("1.23"), types=keys.DT_DECIMAL32)
print(data)

# output: b'%\x00\x02\x00\x00\x00{\x00\x00\x00'

re = io.loads(data)
print(re)

# output: Decimal('1.23')

Serialize Decimal data with both type and scale specified:

data = io.dumps(Decimal("1.23"), types=[keys.DT_DECIMAL32, 4])
print(data)

# output: b'%\x00\x04\x00\x00\x00\x0c0\x00\x00'

re = io.loads(data)
print(re)

# output: Decimal('1.2300')

Example 3

Serialize a vector and convert back:

import dolphindb.settings as keys
import numpy as np
data = io.dumps([1, 2, 3], types=keys.DT_INT)
print(data)

# output: b'\x04\x01\x03\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'

re = io.loads(data)
print(re)

# output: [1, 2, 3]

Serialize an AnyVector and convert back:

data = io.dumps([[1, 2], [3]], types=keys.DT_ANY)
print(data)

# output: b'\x19\x01\x02\x00\x00\x00\x01\x00\x00\x00\x05\x01\x02\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x05\x01\x01\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'

re = io.loads(data)
print(re)

# output: [array([1, 2], dtype=int64), array([3], dtype=int64)]

Serialize an ArrayVector and convert back:

data = io.dumps([[1, 2], [3]], types=keys.DT_INT_ARRAY)
print(data)

# output: b'D\x01\x02\x00\x00\x00\x03\x00\x00\x00\x02\x00\x01\x00\x02\x01\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'

re = io.loads(data)
print(re)

# output: [array([1, 2]) array([3])]

Example 4

Serialize a dictionary and convert back:

data = io.dumps({'a': 1, 'b': 2}, types=[keys.DT_STRING, keys.DT_INT])
print(data)

# output: b'\x04\x05\x12\x01\x02\x00\x00\x00\x01\x00\x00\x00b\x00a\x00\x04\x01\x02\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00'

re = io.loads(data)
print(re)

# output: {'a': 1, 'b': 2}

Serialize a set and convert back:

data = io.dumps({1, 2}, types=keys.DT_INT)
print(data)

# output: b'\x04\x04\x04\x01\x02\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00'

re = io.loads(data)
print(re)

# output: {1, 2}

Example 5

Serialize a table and convert back:

import pandas as pd
from decimal import Decimal
df = pd.DataFrame({
     'a': ["a", "b", "c"],
     'b': [1, 2, 3],
     'c': [Decimal("1.1"), Decimal("2.12"), None],
 })

print(df)

""" output:
   a  b     c
0  a  1   1.1
1  b  2  2.12
2  c  3  None
"""
data = io.dumps(df, types={'a': keys.DT_STRING, 'b': keys.DT_INT, 'c': [keys.DT_DECIMAL32, 3]})
re = io.loads(data)
print(re)

""" output:
   a  b      c
0  a  1  1.100
1  b  2  2.120
2  c  3   None
"""