CSAP Factor Library

The DolphinDB CSAP module mainly includes the following module files:

1. CSAPFactors: Contains 195 standardized factor calculation functions.
2. CSAPPrepare:
   1. Functions starting with prepare: Perform raw data cleaning and structured processing.
   2. Functions starting with calc: Complete the basic indicator calculation, including data cleaning and calculation.
3. CSAPDataSimulation: Generates simulated data required for factor calculation.

All naming rules in the module are based on the definitions of the factors. For example, brandInvest calculates a company's brand investment rate. The input parameters for each factor vary. For details, refer to parameter.csv. Some of the input parameters are as follows:

Place the attached CSAPFactors.dos in the [home]/modules directory. The home directory is set by the configuration parameter homeand can be viewed using the getHomeDir() function. To use the simulated data generation module, place CSAPDataSimulation.dos in the same directory as the CSAPFactors module. To use the data cleaning module, place CSAPPrepare.dos in the same directory as the CSAPFactors module.

For more details on module usage, see DolphinDB Tutorial: Modules.

The CSAP module uses a total of 14 data tables, including Compustat and CRSP. For a complete list, see CSAPFactorTableInfo.

If you do not have the data, you can use the CSAPDataSimulation function in the CSAPDataSimulation.dos to generate simulated data. The function returns a dictionary where keys are table names and values are table data. When generating simulated data, you need to specify four parameters: permno, gvkey, startYear, and endYear. The first two parameters have been described above. startYearand endYearrepresent the time range of the simulated data.

If you already have data that can be calculated directly, ensure that the field names in the data tables are consistent with the parameter names required by the factors. For information such as the input tables and parameter names required by the factors, see CSAPFactorTableInfo.

After preparing the required data sources, some special tables (such as SignalMasterTable) require multi-table joins, and different tables require different data cleaning steps. For convenience, CSAPPrepare.dos helps clean the data, making it easier to proceed with factor calculation. CSAPPrepare.dos defines a corresponding data cleaning function for each table. You can call the data cleaning functions based on the tables required for factor calculation.

The following example loads the m_aCompustat data and performs data cleaning.

use CSAPDataSimulation
use CSAPPrepare

gvkeyList = 10970 10910
startYear = 1987
endYear = 2023
//Data simulation and acquisition
result = CSAPDataSimulation::CSAPDataSimulation(gvkeyList, startYear, endYear)
CCMLinkingTable = result.CCMLinkingTable
CompustatAnnual = result.CompustatAnnual
//Data cleaning
m_aCompustat = CSAPPrepare::prepareM_aCompustat(CompustatAnnual, CCMLinkingTable)

All factors in the CSAPFactors.dos use a vectorized parameter design. You should prepare the dataset required by the target factor according to Section 2.2, and confirm the specific parameter requirements by referring to CSAPFactorTableInfo. The parameters required by different factors vary. Therefore, it is recommended to validate parameter matching before calculation.

Some factors can be calculated using the company's financial data, while others require market data. For convenience, the CSAP module provides two calculation methods:

1. Full calculation: Use the preparexxx functions to perform data cleaning. Then call the calcxxx functions to complete the calculation. For example, call the prepareM_aCompustat function to clean and merge two table datasets, and then call the calcPctTotAcc function to calculate the pctTotAcc factor. After executing the calcPctTotAcc function, the output contains three columns: permno (security identifier), time_avail_m (data timestamp), and PctTotAcc (factor value).

   use CSAPPrepare
   // End-to-end example: data preparation & factor calculation
   cleaned_data = CSAPPrepare::prepareM_aCompustat(CompustatAnnual, CCMLinkingTable)
   result = CSAPPrepare::calcPctTotAcc(
       cleaned_data, 
    startTime=1986.03M, // Standardized format for temporal parameters
       endTime=2010.12M
   )

2. Direct Calculation: If you already have standardized data, you can call factor functions directly: This method requires that input parameters strictly match the function definition; we recommend verifying parameter completeness against the metadata table.

   use CSAPFactors
   // Call factor functions directly (ensure the input data is standardized)
   result = select 
       permno, 
       time_avail_m, 
       pctTotAcc(ni, prstkcc, sstk, dvt, oancf, fincf, ivncf) as PctTotAcc 
   from cleaned_data

After you prepare all base data tables, including core data sources such as Compustat financial statements, CRSP market data, and Fama-French three factors (refer to the data source list in the appendix), you can use the following standardized workflow to automatically calculate all 195 factors. varDict stores all parameters; simply replace the values in the dictionary with the corresponding data sources, and all factors can be dynamically computed through parseExpr.

// Full calculation
use CSAPPrepare
varDict = dict(
    ["startTime", "endTime", "m_aCompustat", "SignalMasterTable", 
     "monthlyCRSP", "monthlyFF", "monthlyLiquidity", "a_aCompustat",
     "m_QCompustat", "CompustatPensions", "CRSPdistributions", "monthlyMarket"], 
    [2005.01M , 2011.12M, m_aCompustat, SignalMasterTable, 
     monthlyCRSP, monthlyFF, monthlyLiquidity, a_aCompustat,
     m_QCompustat, CompustatPensions, CRSPdistributions, monthlyMarket]
)


funcList = select name, syntax from defs("CSAPPrepare::calc%")
resultDict = dict(STRING,ANY)
for (func in funcList) {
    try{
        factor_name = func.name.split("::")[1]
        resultDict[factor_name] = parseExpr(func.name + func.syntax, varDict = varDict).eval()
    }catch(ex){
        print func
        print(ex)
    }
}

Data source list: CSAPFactorTableInfo.xlsx

Parameter description: parameter.csv

Factor module: CSAPFactors.dos

Data cleaning and factor calculation module: CSAPPrepare.dos

Simulated data module: CSAPDataSimulation.dos

The following factor reference table contains explanations and sources for all factors.