QunarOPS / qtdigest

tdigest for python

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

qtdigest

python implementation of Dunning's T-Digest, inspired by nodejs tdigest

Install

pip install qtdigest

Usage

from qtdigest import Tdigest

t = Tdigest()
for i in xrange(1000):
    t.push(random())
P90 = t.percentile(0.9)
print 'P90 = ', P90

API

Tdigest(delta=0.01, K=25, CX=1.1)

  • delta: the compression factor, the max fraction of mass that can be owned by one centroid (bigger, up to 1.0, means more compression).

  • K: a size threshold that triggers recompression as the TDigest grows during input

  • CX: specifies how often to update cached cumulative totals used for quantile estimation during ingest.

  • return: Tdigest instance

Instance of Tdigest

  • push(x, n): add data with value x and weight n

  • size(): return the count of centroids

  • toList(): return the list of all centroids data

  • percentile(p): return the percentage of p(0..1)

  • serialize(): serialize tdigest instance to string, ie: 0.01~25~2~0.00064~0.0013~2~20

  • simpleSerialize(): simply serialize tdigest instance to string, ie: 0.00064~2~0.0013~20

  • deserialize(serialized_str): deserialize the serialized string to tdigest instance. it is a classmethod, so can be called by Tdigest.deserialize(serialized_str)

Performance

platform: MacBook Pro (2.6 GHz Intel Core i5)

data size (push times) cost time
1K 0.07s
10K 0.2s
100K 1.7s
1M 17s

About

tdigest for python


Languages

Language:Python 100.0%