libmc is a memcached client library for Python without any other dependencies in runtime. It's mainly written in C++ and Cython. libmc can be considered as a drop in replacement for libmemcached and python-libmemcached.
libmc is developing and maintaining by Douban Inc. Currently, It is working in production environment, powering all web traffics in douban.com. Realtime benchmark result will be available on travis (soon).
For users:
pip install -e git+https://github.com/douban/libmc.git@master#egg=libmc
Usage:
import libmc
mc = libmc.Client(['localhost:11211', 'localhost:11212'])
mc.set('foo', 'bar')
assert mc.get('foo') == 'bar'
Under the hood, libmc consists of 2 parts: an internal fully-functional memcached client
implementation in C++ and a Cython wrapper around that implementation.
Dynamic memory allocation and memory-copy are slow, so we tried our best to avoid them.
The set_multi
command is not natively supported by the memcached protocol.
Some techniques are applied to make set_multi
command extremely fast in libmc
(compared to some other similiar libraries).
import libmc
mc = libmc.Client(['localhost:11211', 'localhost:11212'], do_split=True, comp_threshold=0, noreply=False,
prefix=None, hash_fn=OPT_HASH_MD5, failover=False)
mc.config(MC_POLL_TIMEOUT, 100) # 100 ms
mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms
mc.config(MC_RETRY_TIMEOUT, 5) # 5 s
-
do_split
: Memcached server will refuse to store value if size >= 1MB, ifdo_split
is enabled, large value (< 10 MB) will be splitted into several blocks. If the value is too large (>= 10 MB), it will not be stored. default:True
-
comp_threshold
: All kinds of values will be encoded into string buffer. Ifbuffer length > comp_threshold > 0
, it will be compressed using zlib. Ifcomp_threshold = 0
, string buffer will never be compressed using zlib. default:0
-
noreply
: Whether to enable memcached'snoreply
behaviour. default:False
-
prefix
: The key prefix. default:''
-
hash_fn
: hashing function for keys. possible values:MC_HASH_MD5
MC_HASH_FNV1_32
MC_HASH_FNV1A_32
MC_HASH_CRC_32
default:
MC_HASH_MD5
NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are per each spec, but they're not compatible with corresponding implementions in libmemcached.
-
failover
: Whether to failover to next server when current server is not available. default:False
-
MC_POLL_TIMEOUT
Timeout parameter used during set/get procedure. (default:300
ms) -
MC_CONNECT_TIMEOUT
Timeout parameter used when connecting to memcached server on initial phase. (default:100
ms) -
MC_RETRY_TIMEOUT
When a server is not available dur to server-end error. libmc will try to establish the broken connection in everyMC_RETRY_TIMEOUT
s until the connection is back to live.(default:5
s)
NOTE: The hashing algorithm for host mapping on continuum is always md5.
Feel free to send a Pull Request. For feature requests or any questions, please open an Issue.
For SECURITY DISCLOSURE, please disclose the information responsibly by sending an email to security@douban.com directly instead of creating a GitHub issue.
No. But if you like, you can write a wrapper for PHP based on the C++ implementation.
No. Only Memcached ASCII protocol is supported currently.
Before libmc, we're using python-libmemcached, which is a python extention for libmemcached. libmemcached is quite weird and buggy. After nearly one decade, there're still some unsolved bugs.
libmc is a single-threaded memcached client. If you initialize a libmc client in one thread but reuse that in another thread,
a Python Exception ThreadUnsafe
will raise in Python.
Yes, with the help of greenify, libmc is friendly to gevent.
Read tests/shabby/gevent_issue.py
for details.
- Thanks to @fahrenheit2539 and the llvm project for the standalone SmallVector implementation.
- Thanks to @miloyip for the high performance i64toa implementation.
- Thanks to Ivan Novikov for the research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS
- Thanks to the PolarSSL project for the md5 implementation.
- Thanks to @lericson for the benchmark script in pylibmc
- Thanks to the libmemcached project and some other projects possibly not mentioned here.
- @mckelvin
- @zzl0
- @windreamer
- @lembacon
Copyright (c) 2014-2015, Douban Inc. All rights reserved.
Licensed under a BSD license: https://github.com/douban/libmc/blob/master/LICENSE