Database Study: Redis vs PostGreSQL

This project is a study on the Redis DataBase Management System (DBMS) and another DBMS, PostGreSQL.
The goal of this study is to shed light on the common points and the differences between the two above DBMS'.
The database used in the study is the animal-crossing database on Kaggle.

Table Of Content.

0 Organization of the repository
1 Prerequisites.
2 Import the database...
- 2.1 ... into PostGreSQL.
- 2.2 ... into Redis.
3 Inspect the tables manually.
4 Execute and time the queries.

0. Organization of the repository [toc]

The repository is organized as follows:
📦 animal-crossing-db-study
┣ 📂 tables -- all the tables from Animal Crossing New Horizons, from Kaggle.
┃ ┣ 📜 accessories.csv
┃ ┣ 📜 achievements.csv
┃ ┣ ...
┃ ┗ 📜 wallpaper.csv
┣ 📂 scripts
┃ ┣ 📜 extract_values.py -- builds a '.json' containing all possible values, by column
┃ ┣ 📂 imports
┃ ┃ ┣ 📜 cact4p.sql -- a .sql script that creates and loads all the tables from tables.
┃ ┃ ┗ 📜 redis_import_main.py -- generates a .redis file or sends tables to a redis server.
┃ ┗ 📂 queries
┃    ┣ 📂 psql
┃    ┃ ┣ 📜 alt-tables-timings.txt -- a summary of timings for altering queries.
┃    ┃ ┣ 📜 queries.sql -- all the queries and their associated questions
┃    ┃ ┣ 📂 query_00
┃    ┃ ┃ ┣ 📜 query.psql -- a query.
┃    ┃ ┃ ┗ 📜 question.txt -- a question.
┃    ┃ ┣ ...
┃    ┃ ┗ 📂 query_38
┃    ┃    ┣ 📜 query.psql -- a query.
┃    ┃    ┗ 📜 question.txt -- a question.
┃    ┗ 📜 redis_queries.py -- the redis version of the queries.
┣ 📜 LICENCE
┣ 📜 README.md
┣ 📜 cact4p -- a bash wrapper to use cact4p.sql more easily.
┣ 📜 seecsv -- a bash tool to manually look at the tables in .csv files.
┣ 📜 relations.md -- a summary of the dataset and the relations between the tables.
┗ 📜 xquery -- a bash tool to execute and time psql queries.

1. Prerequisites. [toc]

Install PostGreSQL: tutorial
Install Redis: tutorial

pip install redis

2. Import the database... [toc]

2.1. ... into PostGreSQL. [toc]

Run the following commands to create a new data base and automatically fill the tables:

createdb animal-crossing

and then

./cact4p tables

cact4p calls the cact4p.sql script which creates and fills all the tables for the user.
"cact4p" stands for Create Animal Crossing's Tables for (4) Postgresql.

2.2. ... into Redis. [toc]

After installing redis server, and redis-py on your computer. And making sure the redis server is running. Execute the python file to generate a redis script.
i.e. in one terminal, or in one tmux-like tool session:

redis-server

and in a another one:

python3 scripts/imports/redis_import_main.py tables

Use python3 scripts/imports/redis_import_main.py -h to learn more about the use of the python script.

3. Inspect the tables manually. [toc]

If you want to inspect the tables in tables, it can be quite tideous in the terminal without the proper tool.
This is why we introduce seecsv, a simple tool to see any .csv table inside the terminal.
Might not work perfectly with large tables, e.g. with pretty long strings, but it can give a good idea of what is inside a given table.
To use it, simply run the following:

./seecsv /path/to/table.csv

One might also want to look at the possible values seen in the tables, e.g. all possible names, all possible colors, ...
To do so, you can use the extract_values.py script in scripts with

python3 scripts/extract_values.py tables

4. Execute and time the queries. [toc]

For the psql part, one can easily execute and time the queries of the repo with the following command:

./xquery scripts/queries/psql/query_<nb>

amtoine / animal-crossing-db-study