A simple Python-based Kinesis Poster and Worker example (aka The Egg Finder)
Poster is a multi-threaded client that creates --poster_count poster
threads to:
- generate random characters, and then
- put the generated random characters into the stream as records
Worker is a thread-per-shard client that:
- gets batches of records, and then
- seeks through the records for the word '
egg' or simply echoes the record to the screen
Multiple Poster or Worker clients can be run simultaneously to generate multi-threaded load on a Kinesis stream.
To get this example working with Python 2.7+, first install boto 2.23+ using:
$ pip install boto
and follow the instructions here to get your credentials setup for use. The credentials you use should permit at least these Kinesis actions: CreateStream, DescribeStream, GetRecords, GetShardIterator, ListStreams & PutRecord. Both the MergeShards and SplitShard actions are
unused in this example.
Once boto is configured with your credentials run:
$ python poster.py my-first-stream
and the Poster will attempt to create the Kinesis stream named
my-first-stream. In a matter of a few minutes the stream will have been
created and you can again run:
$ python poster.py my-first-stream
again. The Poster will then use multiple threads to pump records into the stream.
Once the Poster is pumping records into the stream, then run:
$ python worker.py my-first-stream
which will start the Worker. The Worker will then begin reading records from
the my-first-stream Kinesis stream.
For detailed help and configuration options:
python poster.py --help ..or python worker.py --help, respectively.