falkben / steam-to-sqlite

Public Steam app and achievement data in a sqlite database

steam-to-sqlite

Public Steam app and achievement data in a sqlite database

Quick links

The database and deployments are updated automatically through GitHub actions.

SQLite3 File	database.db ⬇️
Datasette deployment	https://steam-to-sqlite.fly.dev/
GraphQL interface	https://steam-to-sqlite.fly.dev/graphql

To preview what's included in the database schema, you can view models.py, which contains the SQLModel representation, or just explore on Datasette.

Data exploration

The following are some plots generated from the data. Follow the Datasette links to view what's most recent.

• Steam games over time (and you thought your Steam library was large!)

  

  Datasette Link

• Games per genre (who knew there were so many indie games?!)

  

  Datasette Link

• Games with full controller support (stand-in for Steam Deck support)

  

  Datasette Link

Hopefully these inspire you to explore the data. If you find something you want to highlight, PRs are welcome!

Install

To install the project locally:

1. Ensure you have Python >= 3.10. pyenv recommended
2. Create and activate a virtual environment: python -m venv .venv && . .venv/bin/activate
3. Install:
   • pip install -r requirements.txt
   • With dev. dependencies: pip install -r requirements.txt -r dev-requirements.txt
4. Install package: pip install -e . or w/ dev dependencies pip install -e ".[dev]"

Manage dependencies

1. install/upgrade pip-tools: pip install pip-tools -U or globally with pipx: pipx install pip-tools

2. Create lock files with:

   pip-compile -o requirements.txt pyproject.toml --quiet && \
   pip-compile --extra dev -c requirements.txt -o dev-requirements.txt pyproject.toml --quiet

3. Upgrade a package:

   pip-compile -o requirements.txt pyproject.toml --quiet --upgrade-package PACKAGE && \
   pip-compile --extra dev -c requirements.txt -o dev-requirements.txt pyproject.toml --quiet

4. Upgrade all packages with:

   pip-compile -o requirements.txt pyproject.toml --quiet --upgrade && \
   pip-compile --extra dev -c requirements.txt -o dev-requirements.txt pyproject.toml --quiet --upgrade

More here: https://github.com/jazzband/pip-tools/

Run

Download a copy of the database and save locally to database.db

To verify installation:

python steam2sqlite/main.py --help
usage: main.py [-h] [-l [LIMIT]]

options:
  -h, --help            show this help message and exit
  -l [LIMIT], --limit [LIMIT]
                        limit runtime (minutes)

To run:

python steam2sqlite/main.py

Due to rate limits on the public Steam api, the program will take several days to iterate over all the Steam apps in the Steam catalog.

Limit the runtime in minutes with the -l or --limit argument:

python steam2sqlite/main.py --limit 1

Will run for 1 minutes and then (hopefully) exit cleanly with a database partially updated.

Migrations

To upgrade db to current migration/revision

alembic upgrade head

Generate a migration/revision

alembic revision --autogenerate -m "MESSAGE"

Examine generated file in migrations/versions. Pay attention to table/column renames (they will be dropped/created in migration file).

Deploy to Fly

The deployment runs automatically with GitHub actions. To manually deploy:

1. install flyctl

2. flyctl auth login

3. Deploy:

   datasette publish fly database.db \
     --install datasette-graphql --install datasette-vega \
     --app steam-to-sqlite \
     --metadata datasette-data/metadata.json