chulwoohan / pyanomaly

Python library for asset pricing

PyAnomaly

PyAnomaly is a comprehensive python library for asset pricing research with a focus on firm characteristic and factor generation. It covers the majority of the firm characteristics published in the literature and contains various analytic tools that are commonly used in asset pricing research, such as quantile portfolio construction, factor regression, and cross-sectional regression. The purpose of PyAnomaly is NOT to generate firm characteristics in a fixed manner. Rather, we aim to build a package that can serve as a standard library for asset pricing research and help reduce non-standard errors.

The current list of the firm characteristics supported by PyAnomaly can be found in Coverage. PyAnomaly is a live project and we plan to add more firm characteristics and functionalities going forward. We also welcome contributions from other scholars.

PyAnomaly is very efficient, comprehensive, and flexible.

Efficiency

PyAnomaly can generate over 200 characteristics from 1950 under one hour (tested on a desktop with 12th Gen Intel(R) Core(TM) i9-12900KS and 128GB RAM and 1GB network). Once the data have been downloaded from WRDS, the processing time is under 15 minutes. To achieve this, PyAnomaly utilizes numba, multiprocessing, and asyncio packages when possible, but not too heavily to maximize readability of the code. The library is slower on the first run due to caching of numba jitted functions.

Comprehensiveness

PyAnomaly supports over 200 firm characteristics published in the literature. It covers most characteristics in Green et al. (2017) and Jensen et al. (2021), except those that use IBES data. It also provides various tools for asset pricing research.

Flexibility

PyAnomaly adopts the object-oriented programming design philosophy and is easy to customize or add functionalities. This means users can easily change the definition of an existing characteristic, add a new characteristic, or change configurations to run the program. For instance, a user can choose whether to update annual accounting variables quarterly (using Compustat.fundq) or annually (using Compustat.funda), or whether to use the latest market equity or the year-end market equity when generating firm characteristics.

Main Features

• Efficient data download from WRDS using asynco.
• Over 200 firm characteristics generation. You can choose which firm characteristics to generate.

• Factor models

  • Fama-French 3-factor model
  • Fama-French 5-factor model
  • Hou-Xue-Zhang 4-factor model
  • Stambaugh-Yuan 4-factor model

• Analytics

  • Cross-sectional regression
  • 1-D sort
  • 2-D sort
  • Rolling regression
  • Quantile portfolio
  • Long-short portfolio
  • Portfolio performance analysis

• Data tools

  • Filtering
  • Winsorizing
  • Trimming
  • Grouping
  • Population
  • And more...

Changelog

v0.9 - 2022.01.15

Initial version.

v0.923 - 2022.01.16

Multiprocessing in datatools.populate() has been updated to increase the speed.

v0.930 - 2022.01.17

The trend factor of Han, Zhou, and Zhu (2016) has been added. We thank Guofu Zhou for this suggestion.

v0.931 - 2022.01.23

A bug of not returning the result in FUNDA.c_ebitda_mev has been fixed.

A new characteristic method for Enterprise multiple (Loughran and Wellman, 2011), c_enterprise_multiple, has been added to FUNDA, as the previous one (c_ebitda_mev) that implements JKP's SAS code uses a different definition from the original definition. This new method uses the original definition.

v1.0 - 2024.02.28 (Major Update)

There are several important updates in this version and some functions are not backward compatible. See the examples in Cookbook for changes.

Major updates

• Performance upgrade

  The library is now significantly faster and more memory efficient.

• panel.FCPanel

  The Panel class has been divided into two classes: Panel class that serves as the base class for panel data analysis and FCPanel class that inherits Panel and serves as the base class for firm characteristics generation. FUNDA, FUNDQ, CRSPM, CRSPD, and Merge now inherit FCPanel instead of Panel.

• characteristics.CRSPDRaw

  Previously, CRSPD.data contained daily crspd data and CRSPD.chars contained monthly firm characteristics. In the new version, a new class CRSPDRaw handles daily crspd data and is a member of CRSPD. CRSPDRaw.data contains daily crspd data and CRSPD.data contains monthly firm characteristics.

• Factor models

  Two new factor models, Fama-French 5-factor and Stambough and Yuan 4-factor models, have been added.

• CRSP-Compustat link

  If a use don't have WRDS subscription for ccmxpf_linktable, PyAnomlay will create a link table internally and use it to map permno and gvkey. Compared to using ccmxpf_linktable, about 13% of gvkey's are different when using the internal link table ('crsp_comp_linktable').

Minor updates

• Default log directory has been added as config.log_dir.
• Float datatype can be configured to float32 using set_config(float_type='float32').
• New file format, parquet, has been added. To change the file format to parquet, do set_config(file_format='parquet'). The default file format is pickle.
• log.set_log_path() has been revised so that it can create a log file automatically from a file name.
• datatools.classify() has been revised so that if the characteristic is a binary variable, the class is either 0 or (number of quantiles - 1). In the previous version, the class was not deterministic.
• jkp.py has been renamed as factors.py.
• analytics.rolling_beta() has been renamed as numba_support.rolling_regression().
• panel.Panel.rolling_beta() has been renamed as panel.Panel.rolling_regression().

• Input arguments have been changed in the following functions.

  • datatools.classify()
  • datatools.trim()
  • datatools.filter()
  • datatools.winsorize()

• A new argument fname has been added to load_data() of FUNDA, FUNDQ, CRSPM, and CRSPD. If funda, fundq, crspm, and crspd data are modified (e.g., cleansed) and saved with different file names, those names can be given to read data from those modified data files.
• mapping.xlsx: New columns, original sample start date (sample_start) and original sample end date (sample_end), have been added.

New functions

• analytics.grs_test(): GRS (Gibbons, Ross, and Shanken, 1989) test.
• config.set_config(): Set library configuration.
• config.get_config(): Get library configuration.
• datatools.apply_to_groups(): Group data and apply a function to each group.
• datatools.apply_to_groups_jit(): Group data and apply a function to each group (jitted version).
• datatools.apply_to_groups_reduce_jit(): Group data and apply a reduce function to each group (jitted version).
• numba_support.roll_sum(): Rolling sum.
• numba_support.roll_mean(): Rolling mean.
• numba_support.roll_std(): Rolling standard deviation.
• numba_support.roll_var(): Rolling variance.
• numba_support.rank(): Rank.
• numba_support.bivariate_regression(): Bivariate regression.
• numba_support.regression(): Multivariate regression.
• numba_support.rolling_regression(): Rolling regression.
• panel.Panel.apply_to_ids(): Apply a function to each id group.
• panel.Panel.apply_to_dates(): Apply a function to each date group.
• wrdsdata.WRDS.create_crsp_comp_linktable(): Create a CRSP-Compustat link table using cusip.
• wrdsdata.WRDS.add_gvkey_to_crsp_cusip(): Add gvkey to m(d)sf and identify primary stocks using internal link table.

Deprecated functions

• characteristics.FUNDA.convert_to_monthly(): Use Panel.populate() instead.
• characteristics.FUNDQ.convert_to_monthly(): Use Panel.populate() instead.
• datatools.filter_n().
• datatools.groupby_apply(): Use datatools.apply_to_groups(), datatools.apply_to_groups_jit(), or datatools.apply_to_groups_reduce_jit().
• datatools.groupby_apply_np(): Use datatools.apply_to_groups(), datatools.apply_to_groups_jit(), or datatools.apply_to_groups_reduce_jit().
• datatools.rolling_apply(): Use datatools.apply_to_groups(), datatools.apply_to_groups_jit(), or datatools.apply_to_groups_reduce_jit().
• datatools.rolling_apply_np(): Use datatools.apply_to_groups(), datatools.apply_to_groups_jit(), or datatools.apply_to_groups_reduce_jit().

Bug fix

• characteristic.FUNDA.c_currat(): A bug of not returning the result has been fixed.
• characteristics.FUNDQ.c_ni_inc8q(): In the previous version, dibq (difference of ibq) was set to nan in the first 4 quarters. This made some valid ni_inc8q in the first 12 quarters become nan. In the new version, we set all nan values of dibq to 0 before calculating ni_inc8q and ni_inc8q is set to nan if dibq is nan. The revised logic does not lose valid ni_inc8q in the first 12 quarters.
• characteristic.CRSPD.zero_trades_21d(): Fixed dividing by 0 when the total turnover is 0. When counting the number of days in a month, only the days when turnover is not nan are counted. Before, all days were counted.
• characteristic.CRSPD.c_zero_trades_126d(): Fixed dividing by 0 when the total turnover is 0.
• characteristic.CRSPD.c_zero_trades_252d(): Fixed dividing by 0 when the total turnover is 0.
• characteristic.CRSPD.c_rmax5_21d(): A bug when there are only a few distinct return values in a month has been fixed. Suppose the return is positive in two days and 0 in the other days. Previously, rmax5_21d was the mean of the two positive returns. In the new version, it is the mean of the two positive returns and three 0 returns. Also, if days of valid returns (not nan) are fewer than or equal to 5, the result is nan.
• characteristic.Merge.age(): In the previous version, age was the max of (funda history, crspm history). This logic can make the age decrease when funda history is missing: if funda data exists from 2000.01 to 2020.12 and crsp data from 2001.01 to 2022.12, the age will decrease in 2021.01. The logic has been revised so that the age doesn't decrease when funda data is missing.
• panel.Panel.rolling(): When lag > 0, shifted rows were not properly removed. This bug has been fixed.

v1.01 - 2024.03.13

Minor bug fix in analytics.time_series_average().