binsmooth
Python implementation of "Better Estimates from Binned Income Data"
Better Estimates from Binned Income Data: Interpolated CDFs and Mean-Matching
Paul T. von Hippel, David J. Hunter, McKalie Drown
Sociological Science
Volume 4, Number 26, Pages 641-655
2017
Originally implemented in the R package binsmooth.
Usage
from binsmooth import BinSmooth
bin_edges = np.array([0, 18200, 37000, 87000, 180000])
counts = np.array([0, 7527, 13797, 75481, 50646, 803])
mean_estimate = 95000
bs = BinSmooth()
bs.fit(bin_edges, counts, m=mean_estimate)
# Print median estimate
print(bs.inv_cdf(0.5))
Installation
Install via pip
pip install binsmooth
pypi page https://pypi.org/project/binsmooth/
Improvements
Better tail estimate by bounded optimisation rather than the adhoc search method found in the R implementation.
More precise inverse CDF by dynamically sampling the CDF. This is done by sampling proportional to the steepness of the CDF i.e. sampling more in areas where the CDF is steeper.
Warnings
Results may not exactly match R binsmooth because of a different approach
to estimating the tail (upper bound).
Furthermore the fit method uses spline_type="PCHIP" by default, which is
scipy's PchipInterpolator [1]. While the R implementation uses the spline
from [2], which can be mimicked by setting spline_type="HYMAN".
References
[1]: Fritsch, F. N. and Carlson, R. E. (1980). Monotone piecewise cubic interpolation. SIAM Journal on Numerical Analysis
[2]: Hyman, J. M. (1983). Accurate monotonicity preserving cubic interpolation. SIAM Journal on Scientific and Statistical Computing