Backup and Restore

To back up a database, you can use function backupDB.

Example. Back up database "testdb" to the directory /home/$USER/backupDB (where $USER is the username):

dbPath="dfs://testdb"
backupDir="/home/$USER/backupDB"
submitJob("backupDB","backup testdb",backupDB,backupDir,dbPath)

To back up a table, you can use function backupTable.

Example. Back up table "quotes_2" of database "testdb" to the target directory /home/$USER/backupTb:

dbPath="dfs://testdb"
tbName=`quotes_2
backupDir="/home/$USER/backupTb"
submitJob("backupTable","backup quotes_2 in testdb",backupTable,backupDir,dbPath,tbName)

To back up partitions of a table, you can specify the paths to the partitions in the form of a vector and pass it to the backup function.

Example. Back up the partitions /Key3/tp/20120101 and /Key4/tp/20120101 of table "quotes_2" in database "testdb" to the directory /home/$USER/backupPar:

dbPath="dfs://testdb"
tbName=`quotes_2
backupDir="/home/$USER/backupPar"
pars=["/Key3/tp/20120101","/Key4/tp/20120101"]
submitJob("backupPartitions","backup some partitions in quotes_2 in testdb",backup,backupDir,dbPath,false,true,true,tbName,pars)

To check the status of backup/restore jobs, you can use function getBackupStatus.

Example. Log in as an admin and check all backup/restore jobs:

login(`admin, `123456)
getBackupStatus()

• An administrator can check the jobs submitted by a specific user if the parameter userName is specified; Otherwise, all jobs are returned.

• A non-administrator can only obtain the status of jobs submitted by the current user.

The function returns:

The table displays information on the databases and tables involved in each job, including the number of partitions, completion percentage, end time (or estimated completion time), etc.

Note:

• Each row in the table represents a backup/restore job on a table.
• The function only checks backup jobs that are executed after the data node is started. When a data node is rebooted, the status of previous backup jobs will be lost.

You can obtain the backup information of a DFS table with getBackupList. The function returns a table where each row represents a backup partition.

Example. Use getBackupList to obtain backup information of table "quotes_2" from database testdb.

dbPath="dfs://testdb"
backupDir="/home/$USER/backupDB"
getBackupList(backupDir,dbPath,`quotes_2)

Part of the table is shown below:

You can obtain the backup information on a certain partition, including the table schema, chunk path, number of rows, ID, and version number. See details of function getBackupMeta.

You can use function checkBackup to verify the checksum of each partition in the backup files. The function returns a table that displays only partitions with checksum mismatch.

Example. Check the data integrity of the backup file from section Back Up a Database.

dbPath="dfs://testdb"
backupDir="/home/$USER/backupDB"
checkBackup(backupDir,dbPath)

Note that the time it takes to verify the checksum is close to the backup time.

In case of checksum mismatch, you can execute backup by specifying the parameter force to true to forcibly back up the specified partitions. If force is set to false, DolphinDB performs an incremental backup which only checks the metadata of partitions, so it may skip partitions with inconsistent data.

To demonstrate the function usage, we will manually generate a backup file with inconsistent checksum:

1. Call the function checkBackup.

2. Remove the backup file and create an empty file with the same name. Then execute checkBackup again.

3. Back up the partitions again, and then execute checkBackup.

We first verify the backup file of "quotes_2" from an OLAP database.

checkBackup(backupDir,dbPath,`quotes_2)

Nothing is returned, meaning the checksum has passed the check.

Then we remove the volume.col file of the partition with chunk ID 086b0c1e-0386-95a4-4b4a-5038f4fd5d51, and add an empty file with the same name. Execute checkBackup again and the result shows the checksum mismatch in the partition:

Next, we can back up the partition again:

backup(backupDir=backupDir,dbPath=dbPath,force=true,parallel=true,snapshot=true,tableName=`quotes_2,partition="/Key4/tp/20120103")

Then checkBackup returns empty, meaning that the checksum now matches. You can use the function to examine whether a backup file is corrupted.

You can check the backup files in two phases:

• After a backup job is finished, check whether the job has been completed properly;

• Before restoring data, check whether a backup file is corrupted.

All these functions should be executed on the node where the backup job is running.

To restore a database, you can use restoreDB.

Example 1. Restore the database backed up in Section Back Up a Database to a new cluster.

dbPath="dfs://testdb"
backupDir="/home/$USER/backupDB"
submitJob("restoreDB","restore testdb in new cluster",restoreDB,backupDir,dbPath)

The above script only specifies the backup directory storing the backup files and the target directory where the restored databases should be placed. Before restoring data to the new cluster, the restoreDB function first creates a database testdb in the target directory.

You can also restore a database from another database within one cluster.

Example 2. Restore the database backed up in Section Back Up a Database to the target database restoredb within one cluster.

dbPath="dfs://testdb"
backupDir="/home/$USER/backupDB"
restoreDBPath="dfs://restoredb"
submitJob("restoreDB2","restore testdb to restoredb in the original cluster",restoreDB,backupDir,dbPath,restoreDBPath)

To restore the data of a table, you can use restoreTable.

Example 1. Restore table quotes_2 backed up in Section Back up a DFS Table to a new cluster:

dbPath="dfs://testdb"
tbName=`quotes_2
backupDir="/home/$USER/backupTb"
submitJob("restoreTable","restore quotes_2 in testdb to new cluster",restoreTable,backupDir,dbPath,tbName)

The above script only specifies the backup directory storing the backup files and the target directory where the restored tables should be placed. The restoreTable function first creates table quotes and database testdb in the new cluster, and restores data to quotes.

You can restore a table to another database within one cluster by specifying the path of the target database.

Example 2. Restore the data backed up in Section Back up a DFS Table to table "quotes_2" of database "restoredb2" within one cluster:

dbPath="dfs://testdb"
tbName=`quotes_2
backupDir="/home/$USER/backupTb"
restoreDBPath="dfs://restoredb2"
submitJob("restoreTable2","restore quotes_2 in testdb to quotes_2 in restoredb",restoreTable,backupDir,dbPath,tbName,restoreDBPath)

You can restore a table to another table within one database by specifying the table name.

Example 3. Restore the data backed up in Section Back up a DFS Table to the table "quotes_restore" of database "testdb".

dbPath="dfs://testdb"
tbName=`quotes_2
backupDir="/home/$USER/backupTb"
restoreTb="quotes_restore"
submitJob("restoreTable3","restore quotes_2 to quotes_restore in testdb",restoreTable,backupDir,dbPath,tbName,,restoreTb)

To restore partitions of a table, you can use function restore. Different from backup, the parameter partition of function restore can only take a scalar with which multiple partitions can be specified using wildcards. For example, you can use %/Key4/tp/% to specify partitions /Key4/tp/20120101, /Key4/tp/20120103, /Key4/tp/20120104, etc.

Example. Restore the partitions backed up in Section Back Up Partitions to a new database.

dbPath="dfs://testdb"
backupDir="/home/$USER/backupPar"
tbName=`quotes_2
pars=["/testdb/Key3/tp/20120101","/testdb/Key4/tp/20120101"]
for (par in pars){
	restore(backupDir,dbPath,tbName,par,false,,true,true)
}

To migrate a cluster, you can back up all databases to a directory, and then restore the cluster with function migrate.

Example. Back up databases "testdb" and "testdb_tsdb" to the target directory, and migrate these databases to a new cluster:

//Back up testdb and tesdb_tsdb
dbPath="dfs://testdb"
dbPath2="dfs://testdb_tsdb"
backupDir="/home/$USER/migrate"
submitJob("backupForMigrate","backup testdb for migrate",backupDB,backupDir,dbPath)
submitJob("backupForMigrate2","backup testdb_tsdb for migrate",backupDB,backupDir,dbPath2)

//Restore these databases to a new cluster
backupDir="/home/$USER/migrate"
submitJob("migrate","migrate testdb and testdb_tsdb to new cluster",migrate,backupDir)

migrate is similar to restoreDB, and the differences lie in:

• You can use migrate to restore multiple databases in batches, whereas restoreDB can only restore a database at a time.
• When the target databases/tables share the same names as the originals, migrate can be used only when the originals are deleted, while restoreDB does not have such limitation. Therefore, migrate does not support incremental recovery or interrupted recovery resumption, and it is usually used to migrate a newly-deployed cluster.

You can also use function getBackupStatus to check whether a restore job has been completed. See Section Check Backup Jobs.