sqlDS

sqlDS(sqlObj, [forcePartition=false])

sqlObj is SQL metacode. For more details about metacode please refer to the section of Metaprogramming.

forcePartition (optional) is a Boolean value. The default value is false. If it is set to false, the system checks if the query can be splitted into multiple child queries. If not then it won't split the query over partitions and will throw an exception. However, when it is set to true, the system splits the query over partitions and runs the query on selected partitions. Cases when a query cannot be splitted into child queries include (1) the group by columns are not partitioning columns; (2) the order by clause is specified, among others.

Create a list of data sources based on the input SQL metacode. If the table in the SQL metacode has n partitions, sqlDS generates n data sources. If the SQL metacode doesn't contain any partitioned table, sqlDS returns a tuple containing one data source. This function divides a large file into several partitions, each representing a subset of the data, and returns a tuple of the data source. A data source is generally represented by a code object and serves as a function call that takes metaprogramming as its parameter and returns a table.

n=1000000
date=take(2019.01.01..2019.01.03,n)
sym = take(`C`MS`MS`MS`IBM`IBM`IBM`C`C$SYMBOL,n)
price= take(49.6 29.46 29.52 30.02 174.97 175.23 50.76 50.32 51.29,n)
qty = take(2200 1900 2100 3200 6800 5400 1300 2500 8800,n)
t=table(date, sym, price, qty)

db1=database("",VALUE,2019.01.01..2019.01.03)
db2=database("",VALUE,`C`MS`IBM)
db=database("dfs://stock",COMPO,[db1,db2])
trades=db.createPartitionedTable(t,`trades,`date`sym).append!(t)

ds=sqlDS(<select * from trades where date=2019.01.02>);

typestr ds;
// output
ANY VECTOR

size ds;
// output
3

ds[0];
// output
DataSource< select [7] * from trades [partition = /stock/20190102/C] >

ds[1];
// output
DataSource< select [7] * from trades [partition = /stock/20190102/IBM] >

ds[2];
// output
DataSource< select [7] * from trades [partition = /stock/20190102/MS] >