HDF5

The DolphinDB hdf5 plugin imports HDF5 datasets into DolphinDB and supports data type conversions.

Installation (with installPlugin)

Required server version: DolphinDB 2.00.10 or higher

OS: Windows and Linux x86-64

Installation Steps:

(1) Use listRemotePlugins to check plugin information in the plugin repository.

Note: For plugins not included in the provided list, you can install through precompiled binaries or compile from source. These files can be accessed from our GitHub repository by switching to the appropriate version branch.

login("admin", "123456")
listRemotePlugins()

(2) Invoke installPlugin for plugin installation

installPlugin("hdf5")

(3) Use loadPlugin to load the plugin before using the plugin methods.

loadPlugin("hdf5")

Method References

ls

Syntax

ls(fileName)

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.

Details

List all the objects (including datasets and groups) and their types in an HDF5 file. The method returns the number of columns and rows of a dataset. For example, DataSet{(7,3)} represents 7 columns and 3 rows.

Example

hdf5::ls("/smpl_numeric.h5")

/* output:
        objName	     objType
        --------------------
        /            Group
        /double	     DataSet{(7,3)}
        /float	     DataSet{(7,3)}
        /schar	     DataSet{(7,3)}
        /sint	     DataSet{(7,3)}
        /slong	     DataSet{(7,3)}
        /sshort	     DataSet{(7,3)}
        /uchar	     DataSet{(7,3)}
        /uint	     DataSet{(7,3)}
        /ulong	     DataSet{(1,1)}
        /ushort	     DataSet{(7,3)}
*/

hdf5::ls("/named_type.h5")

/* output:
        objName      objType
        ----------------------
        /            Group      
        /type_name   NamedDataType
*/

lsTable

Syntax

lsTable(fileName)

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.

Details

List all the table information in an HDF5 file, i.e., HDF5 dataset information, including table name, dimension, and type.

Example

hdf5::lsTable("/smpl_numeric.h5")

/* output:
       tableName    tableDims	 tableType
       /double        7,3       H5T_NATIVE_DOUBLE
       /float	      7,3       H5T_NATIVE_FLOAT
       /schar	      7,3       H5T_NATIVE_SCHAR
       /sint	      7,3       H5T_NATIVE_INT
       /slong	      7,3       H5T_NATIVE_LLONG
       /sshort	      7,3       H5T_NATIVE_SHORT
       /uchar	      7,3       H5T_NATIVE_UCHAR
       /uint	      7,3       H5T_NATIVE_UINT
       /ulong	      1,1       H5T_NATIVE_ULLONG
       /ushort	      7,3       H5T_NATIVE_USHORT
*/

extractHDF5Schema

Syntax

extractHDF5Schema(fileName, datasetName)

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.
  • datasetName: A STRING scalar indicating the dataset name, i.e., the table name. It can be obtained using ls or lsTable.

Details

Generate the schema table for the specified dataset in the HDF5 file. The schema table contains 2 columns: column names and data types.

Example

hdf5::extractHDF5Schema("/smpl_numeric.h5","sint")

/* output:
        name	type
        col_0	INT
        col_1	INT
        col_2	INT
        col_3	INT
        col_4	INT
        col_5	INT
        col_6	INT
*/

hdf5::extractHDF5Schema("/compound.h5","com")

/* output:
        name	type
        fs	STRING
        vs	STRING
        d	DOUBLE
        t	TIMESTAMP
        l	LONG
        f	FLOAT
        i	INT
        s	SHORT
        c	CHAR
*/

loadHDF5

Syntax

loadHDF5(fileName,datasetName,[schema],[startRow],[rowNum])

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.
  • datasetName: A STRING scalar indicating the dataset name, i.e., the table name. It can be obtained using ls or lsTable.
  • schema (optional): A table containing column names and types. To modify the data type of a column that is automatically determined by the system, the schema table needs to be modified and used as a parameter in loadHdf5.
  • startRow (optional): An integer indicating the start row from which to read the HDF5 dataset. If not specified, the dataset will be read from the beginning.
  • rowNum (optional): An integer indicating the number of rows to read from the HDF5 dataset. If not specified, the reading continues until the end of the dataset.

Details

  • Load an HDF5 file into a DolphinDB in-memory table. rowNum specifies the number of rows to read from the HDF5 dataset, instead of the output DolphinDB table. For supported data types and data conversion rules, refer to the "Data Type Mappings" section.

Example

hdf5::loadHDF5("/smpl_numeric.h5","sint")

/* output:
        col_0	col_1	col_2	col_3	col_4	col_5	col_6
        (758)	8	(325,847)	87	687	45	90
        61	0	28	77	546	789	45
        799	5,444	325,847	678	90	54	0
*/


scm = table(`a`b`c`d`e`f`g as name, `CHAR`BOOL`SHORT`INT`LONG`DOUBLE`FLOAT as type)
hdf5::loadHdf5("../hdf5/h5file/smpl_numeric.h5","sint",scm,1,1)

/* output:
        a	b	c	d	e	f	g
        '='	false	28	77	546	789	45
*/

Note: The dimension of the dataset must be 2 or less. Only 2D or 1D tables can be parsed.

loadPandasHDF5

Syntax

loadPandasHDF5(fileName,groupName,[schema],[startRow],[rowNum])

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.
  • groupName: The identifier of the group, i.e. the key name.
  • schema (optional): A table containing column names and types. To modify the data type of a column that is automatically determined by the system, the schema table needs to be modified and used as a parameter in loadHdf5.
  • startRow (optional): An integer indicating the start row from which to read the HDF5 dataset. If not specified, the dataset will be read from the beginning.
  • rowNum (optional): An integer indicating the number of rows to read from the HDF5 dataset. If not specified, the reading continues until the end of the dataset.

Details

Load an HDF5 file saved by Pandas into a DolphinDB in-memory table. rowNum specifies the number of rows to read from the HDF5 dataset, instead of the output DolphinDB table. For supported data types and data conversion rules, refer to the "Data Type Mappings" section.

Example

hdf5::loadPandasHDF5("/home/ffliu/Data/data.h5","/s",,1,1)

/* output:
        A	 B	C  D  E
        28 77	54 78 9
*/

loadHDF5Ex

Syntax

loadHDF5Ex(dbHandle,tableName,[partitionColumns],fileName,datasetName,[schema],[startRow],[rowNum],[transform])

Parameters

  • dbHandle: The database handle specified to save the input data into a distributed database.
  • tableName: The table name specified to save the input data into a distributed database.
  • partitionColumns (optional): A STRING scalar/vector indicating partitioning column(s), which needs to be specified when the database is not sequentially (SEQ) partitioned. In composite partitioning, partitionColumns is a STRING vector.
  • fileName: A STRING scalar indicating the HDF5 file name.
  • datasetName: A STRING scalar indicating the dataset name, i.e., the table name. It can be obtained using ls or lsTable.
  • schema (optional): A table containing column names and types. To modify the data type of a column that is automatically determined by the system, the schema table needs to be modified and used as a parameter in loadHdf5.
  • startRow (optional): An integer indicating the start row from which to read the HDF5 dataset. If not specified, the dataset will be read from the beginning.
  • rowNum (optional): An integer indicating the number of rows to read from the HDF5 dataset. If not specified, the reading continues until the end of the dataset.
  • transform (optional): An unary function that takes a table as the parameter. If specified, it is necessary to create a partitioned table before loading the data. The system will apply the function specified by transform to the data in the HDF5 file and save the results into the database.

Details

Convert the datasets in an HDF5 file into a DolphinDB distributed table. The metadata of the table is loaded into the memory. rowNum specifies the number of rows to read from the HDF5 dataset, instead of the output DolphinDB table. For supported data types and data conversion rules, refer to the "Data Type Mappings" section.

Example

  • SEQ partitioned table on disk

    db = database("seq_on_disk", SEQ, 16)
    hdf5::loadHDF5Ex(db,`tb,,"/large_file.h5", "large_table")
  • SEQ partitioned table in memory

    db = database("", SEQ, 16)
    hdf5::loadHDF5Ex(db,`tb,,"/large_file.h5", "large_table")
  • Non-SEQ partitioned table on disk

    db = database("non_seq_on_disk", RANGE, 0 500 1000)
    hdf5::loadHdf5Ex(db,`tb,`col_4,"/smpl_numeric.h5","sint")
  • Non-SEQ partitioned table in memory

    db = database("", RANGE, 0 500 1000)
    t0 = hdf5::loadHDF5Ex(db,`tb,`col_4,"/smpl_numeric.h5","sint")
  • Specify parameter transform to convert numeric date and time values (e.g., 20200101) to a DATE type (e.g., 2020.01.01).

    dbPath="dfs://DolphinDBdatabase"
    db=database(dbPath,VALUE,2020.01.01..2020.01.30)
    dataFilePath="/transform.h5"
    datasetName="/SZ000001/data"
    schemaTB=hdf5::extractHDF5Schema(dataFilePath,datasetName)
    update schemaTB set type="DATE" where name="trans_time"
    tb=table(1:0,schemaTB.name,schemaTB.type)
    tb1=db.createPartitionedTable(tb,`tb1,`trans_time);
    def i2d(mutable t){
        return t.replaceColumn!(`trans_time,datetimeParse(string(t.trans_time),"yyyyMMdd"))
    }
    t = hdf5::loadHDF5Ex(db,`tb1,`trans_time,dataFilePath,datasetName,,,,i2d)

HDF5DS

Syntax

HDF5DS(fileName,datasetName,[schema],[dsNum])

Parameters

  • fileName: A STRING scalar indicating the HDF5 file name.
  • datasetName: A STRING scalar indicating the dataset name, i.e., the table name. It can be obtained using ls or lsTable.
  • schema (optional): A table containing column names and types. To modify the data type of a column that is automatically determined by the system, the schema table needs to be modified and used as a parameter in loadHdf5.
  • dsNum (optional): The number of data sources to be generated. HDF5DS divides the whole table equally into dsNum tables. If not specified, it will generate one data source.

Details

Generate a tuple of data sources according to the input file name and dataset name.

Example

ds = hdf5::HDF5DS(smpl_numeric.h5","sint")
size ds;
// output:1
ds[0];
//output:DataSource< loadHDF5("/smpl_numeric.h5", "sint", , 0, 3) >

ds = hdf5::HDF5DS(smpl_numeric.h5","sint",,3)
size ds;
// output:3
ds[0];
// output:DataSource< loadHDF5("/smpl_numeric.h5", "sint", , 0, 1) >
ds[1];
// output:DataSource< loadHDF5("/smpl_numeric.h5", "sint", , 1, 1) >
ds[2];
// output:DataSource< loadHDF5("/smpl_numeric.h5", "sint", , 2, 1) >

Note: HDF5 does not support concurrent reads, for example:

ds = hdf5::HDF5DS("/smpl_numeric.h5", "sint", ,3)
res = mr(ds, def(x) : x)

To correct this error, set parameter parallel of function mr to false:

ds = hdf5::HDF5DS("/smpl_numeric.h5", "sint", ,3)
res = mr(ds, def(x) : x,,,false)

saveHDF5

Syntax

saveHDF5(table, fileName, datasetName, [append], [stringMaxLength])

Parameters

  • table: The table to be saved.
  • fileName: A STRING scalar indicating the HDF5 file name.
  • datasetName: A STRING scalar indicating the dataset name, i.e., the table name. It can be obtained using ls or lsTable.
  • append (optional): A BOOL value indicating whether to append data to an existing table. The default value is false.
  • stringMaxLength (optional): A numeric value indicating the maximum length of strings, which only applies to the data of STRING and SYMBOL type in the table. The default value is 16.

Details

Save the DolphinDB in-memory table to a specified dataset in an HDF5 file. For supported data types and data conversion rules, refer to the "Data Type Mappings" section.

Example

hdf5::saveHDF5(tb, "example.h5", "dataset name in hdf5")

Note:

  • NULL values are not supported in HDF5 files. If there are NULL values in DolphinDB tables, they will be saved as the default value defined in the "Data Type Mappings" section.

  • To read the h5 files generated by the hdf5 plugin through python, you can use the h5py library as follows:

    import h5py
    f = h5py.File("/home/workDir/dolphindb_src/build/test.h5", 'r')
    print(f['aaa']['TimeStamp'])
    print(f['aaa']['StockID'])

Data Type Mappings

The floating point and integer types in the HDF5 file will be converted to H5T_NATIVE_* type (via H5Tget_native_type).

Integer

Type in HDF5Default Value in HDF5Type in CType in DolphinDB
H5T_NATIVE_CHAR'\0'signed char / unsigned charchar/short
H5T_NATIVE_SCHAR'\0'signed charchar
H5T_NATIVE_UCHAR'\0'unsigned charshort
H5T_NATIVE_SHORT0shortshort
H5T_NATIVE_USHORT0unsigned shortint
H5T_NATIVE_INT0intint
H5T_NATIVE_UINT0unsigned intlong
H5T_NATIVE_LONG0longint/long
H5T_NATIVE_ULONG0unsigned longunsupported/long
H5T_NATIVE_LLONG0long longlong
H5T_NATIVE_ULLONG0unsigned long longunsupported
  • In DophinDB, all numeric types are signed. To prevent overflow, all unsigned types, except for 64-bit unsigned types, are converted to high-order signed types. Specifically, 64-bit unsigned types are converted to 64-bit signed types. If an overflow occurs during the conversion, the value is converted to the maximum value of the 64-bit signed type.
  • H5T_NATIVE_CHAR corresponds to the char type in C. Whether char is signed or unsigned depends on the compiler and platform. Signed char is converted to the CHAR type in DolphinDB and unsigned char is converted to the SHORT type.
  • H5T_NATIVE_LONG and H5T_NATIVE_ULONG correspond to the long type in C.
  • All integer types can be converted to the numeric types (BOOL, CHAR, SHORT, INT, LONG, FLOAT, and DOUBLE) in DolphinDB. An overflow may occur during the conversion. For example, the conversion from LONG to INT returns the maximum or minimum value of INT.

Floating Point

Type in HDF5Default Value in HDF5Type in CType in DolphinDB
H5T_NATIVE_FLOAT+0.0ffloatfloat
H5T_NATIVE_DOUBLE+0.0doubledouble
  • IEEE754 floating point types are all signed.
  • All floating point types can be converted to the numeric types (BOOL, CHAR, SHORT, INT, LONG, FLOAT, and DOUBLE) in DolphinDB. An overflow may occur during the conversion. For example, the conversion from DOUBLE to FLOAT returns the maximum or minimum value of FLOAT.

Temporal

Type in HDF5Default Value in HDF5Type in CType in DolphinDB
H5T_UNIX_D32BE1970.01.01T00:00:004 bytes integerDT_TIMESTAMP
H5T_UNIX_D32LE1970.01.01T00:00:004 bytes integerDT_TIMESTAMP
H5T_UNIX_D64BE1970.01.01T00:00:00.0008 bytes integerDT_TIMESTAMP
H5T_UNIX_D64LE1970.01.01T00:00:00.0008 bytes integerDT_TIMESTAMP
  • The predefined temporal types of HDF5 are 32-bit or 64-bit POSIX time. Due to the lack of an official definition for temporal types in HDF5, this plugin interprets 32-bit temporal types as the number of seconds since 1970 and 64-bit temporal types as milliseconds. All temporal types are uniformly converted by the plugin into a 64-bit integer and then converted to the timestamp type in DolphinDB.
  • The aforementioned temporal types can be converted to time-related types in DolphinDB: DATE, MONTH, TIME, MINUTE, SECOND, DATETIME, TIMESTAMP, NANOTIME, NANOTIMESTAMP.

String

Type in HDF5Default Value in HDF5Type in CType in DolphinDB
H5T_C_S1""char*DT_STRING
  • H5T_C_S1 includes fixed-length strings and variable-length strings.
  • The string type can be converted to string-related types in DolphinDB: STRING and SYMBOL.

Enum

Type in HDF5Corresponding C TypeCorresponding DolphinDB Type
ENUMenumDT_SYMBOL
  • The enum type will be converted to a SYMBOL variable in DolphinDB. Note that the enum value and the order of size of each string will not be saved. For example, an enum value HDF5_ENUM{"a"=100,"b"=2000,"c"=30000} might be converted to SYMBOL{"a"=3,"b"=1"c"=2}.
  • The enum type can be converted to string-related types in DolphinDB: STRING and SYMBOL.

Compound and Array

Type in HDF5Corresponding C TypeCorresponding DolphinDB Type
H5T_COMPOUNDstruct\
H5T_ARRAYarray\
  • Compound and array types can be parsed as long as they do not contain unsupported types. Nested data types can also be parsed.
  • The conversion of complex types depends on their internal subtypes.

Table Structure

Simple Data Type Table Structure

The tables containing simple data types in HDF5 remain the same structure after being imported into DolphinDB.

Simple Table in HDF5

12
1int(10)int(67)
2int(20)int(76)

Simple Table in DolphinDB

col_1col_2
11067
22076

Complex Data Type Table Structure

For tables containing complex data types in HDF5, their structure after being imported into DolphinDB depends on their specific data type.

Table of Compound Data in HDF5

12
1structstruct
2structstruct

Table of Compound Data in DolphinDB

abc
1123.7
2112131.7
3122232.7
4132333.7

Table of Arrays in HDF5

12
1array(1,2,3)array(4,5,6)
2array(8,9,10)array(15,16,17)

Table of Arrays in DolphinDB

array_1array_2array_3
1123
2456
38910
4151617

Table of Nested Data in HDF5

For tables containing nested data types in HDF5, a prefix "A" is added to the converted tables in DolphinDB to represent arrays and a prefix "C" is added to represent compound data types.

12
1struct{a:array(1,2,3) b:2 c:struct{d:"abc"}}struct{a:array(7,8,9) b:5 c:struct{d:"def"}}
2struct{a:array(11,21,31) b:0 c:struct{d:"opq"}}struct{a:array(51,52,53) b:24 c:struct{d:"hjk"}}

Table of Nested Data in DolphinDB

Aa_1Aa_2Aa_3bCc_d
11232abc
27895def
31121310opq
451525324hjk

Performance

Environment

  • CPU: i7-7700 3.60GHZ
  • SSD: sequential read with a speed up to 460~500MB/S

Dataset import performance

  • Int
    • Row number: 1024 * 1024 * 16
    • Column number: 64
    • File size: 4G
    • Elapsed time: 8s
  • Unsigned int
    • Row number: 1024 * 1024 * 16
    • Column number: 64
    • File size: 4G
    • Elapsed time: 9s
  • Variable-length string
    • Row number: 1024 * 1024
    • Column number: 64
    • File size: 3.6G
    • Elapsed time: 17s
  • Compound
    • Row number: 1024 * 1024 * 62
    • Column number: 9 for subtypes (STR, STR, DOUBLE, INT, LONG, FLOAT, INT, SHORT, CHAR)
    • File size: 3.9G
    • Elapsed time: 10s
  • compound array
    • Row number: 1024 * 128 * 62
    • Column number: 8 * 9 for subtypes (STR, STR, DOUBLE, INT, LONG, FLOAT, INT, SHORT, CHAR)
    • File size: 3.9G
    • Elapsed time: 15s