sqlDS#

swordfish.function.sqlDS()#

Create a list of data sources based on the input SQL metacode. If the table in the SQL metacode has n partitions, sqlDS generates n data sources. If the SQL metacode doesn’t contain any partitioned table, sqlDS returns a tuple containing one data source. This function divides a large file into several partitions, each representing a subset of the data, and returns a tuple of the data source. A data source is generally represented by a code object and serves as a function call that takes metaprogramming as its parameter and returns a table.

Parameters:

  • sqlObj (Constant) – SQL metacode. For more details about metacode please refer to the section of Metaprogramming.

  • forcePartition (Constant, optional) –

    A Boolean value, by default DFLT. If it is set to false, the system checks if the query can be splitted into multiple child queries. If not then it won’t split the query over partitions and will throw an exception.

    However, when it is set to true, the system splits the query over partitions and runs the query on selected partitions. Cases when a query cannot be splitted into child queries include (1) the group by columns are not partitioning columns; (2) the order by clause is specified, among others.

On this page