How to manage Python environments

[utterances bot]

How to manage Python environments

Here is an excerpt from a Python working session that I led about managing Python environments using virtualenv, make, and pip-tools. Found it a bit useful s...

https://ljvmiranda921.github.io/notebook/2021/05/12/how-to-manage-python-envs/

[nhuber]

In your first makefile, two of the later targets have some "prefaces" (format, test).

    venv/bin/pre-commit run --all-files

test: dev ## Run all tests with coverage
    venv/bin/pytest tests --cov=src -v --cov-report=term-missing```

What are those doing?

What's the diff between requirements.in and requirements.txt? (Why not just put everything in a requirements.txt?)

[Lj Miranda]

Awesome to see you here @nhuber!

What's the diff between requirements.in and requirements.txt? (Why not just put everything in a requirements.txt?)

The file requirements.txt is generated from requirements.in after resolving dependencies (we resolve dependencies to avoid dependency hell). The former (.txt) is computer-friendly, whereas the latter (.in) is human-friendly.

• requirements.in: human-friendly. I can just specify that "I need numpy, the 1.13 version of pytorch, and pandas". This is what you'll usually update as a developer

# sample requirements.in
# "just give me numpy and pandas that are compatible with this version of pytorch"
numpy 
pandas
pytorch==1.13

• requirements.txt: computer-friendly, because the exact pinned versions of a library is specified (no dependency hell). Humans don't need to touch this in our workflow, just need to generate

# sample requirements.txt
# after dependency resolution:
numpy==1.14
pandas==1.25.2
pytorch==1.13

Then it's the requirements.txt that we will install in our environment

[Lj Miranda]

In your first makefile, two of the later targets have some "prefaces" (format, test).
What are those doing?

So the good thing about Makefiles is that they kinda act like a DAG (although don't overdo it, I prefer Makefiles to be as simple as possible)

If you meant the dev keyword, then it means that make will first run the dev target before running what's inside format.

So in this case, if we enter:

make format

Then make will run the dev target first, which contains the following comand:

venv/bin/pip-sync requirements.txt
venv/bin/pre-commit install  # (out-of-scope for this session)

Then proceed to running the actual format command:

venv/bin/pre-commit run --all-files