Experiments with Redis Bloom and the Text from "Moby Dick"

This repository contains a small example using the text from the book "Moby Dick" as a data source to compare and contrast the benefits and tradeoffs of using different Redis data structures. We'll use the deterministic Set data structure, and three probabilistic data structures: Hyperloglog, Bloom Filter and Top-K. These probabilistic data structures generally use hashing to be more memory efficient at the cost of some accuracy and the ability to recall data added to them.

The Set and Hyperloglog are built into core Redis, to use the Bloom Filter and Top K data structures, you'll need Redis Stack or the RedisBloom module.

Prerequisites

To try out this project yourself, you'll need:

Redis Stack (installed locally, available free in the cloud, or use the supplied Docker Compose file).
Docker Desktop (if using the Docker Compose file).
git command line tools.
Node.js version 14.8 or higher (I've tested the code using Node.js version 16.4.2).
OR
Python version 3.6 or higher (I've tested the code using Python version 3.10.6).

Get the Code

To get the code, clone the repo to your machine:

$ git clone https://github.com/redis-developer/redisbloom-moby-dick.git
$ cd redisbloom-moby-dick/

Application Setup

This project has two versions, one written in NodeJS and other in Python. For the NodeJS version, head to the nodejs directory by doing:

$ cd nodejs/

and follow the instructions given in the README file inside the directory.

For the Python version, head to the python directory by doing:

$ cd python/

and follow the instructions given in the README file inside the directory.

Further Exercises

Try using the following commands to check if a given word is in the Set, or possibly in the Bloom Filter:

SISMEMBER: Find out for sure whether a word is in the Set whose key is mobydick:words:set. Try checking multiple words at once with SMISMEMER
BF.EXISTS: Find out whether a word may be or is absolutely not in the Bloom Filter whose key is mobydick:words:bloom. Try checking multiple words at once with BF.MEXISTS

Why can't we say with absolute certainty that a word is in the Bloom Filter?

Licensing

The code is made available under the terms of the MIT license. I obtained the original text file of the book "Moby Dick" from Project Gutenberg. This text is in the public domain and available in its original form here.

radioactive11 / redisbloom-moby-dick