An error occurred while using the Python magic library at Docker & Github Action. "E ModuleNotFoundError: No module named 'magic'"

[Charm AI]

🐛 Issue Description:
Running a Python project using the python-magic library within a Docker container on Windows 10 OS results in the error "Module magic not found."

🔧 Proposed Fix:
Initially, changing the dependency from python-magic to python-magic-bin resolved the issue in the local environment. However, when running the project in a Docker container, the same error resurfaced unexpectedly.

🐋 Dockerfile Content:

# Dockerfile for API service
FROM python:3.11.0 as base

# Dependencies installations
RUN apt-get update && apt-get install -y \
    poppler-utils \
    tesseract-ocr \
    libgl1-mesa-glx \
    libmagic1 \
    python3-magic

# Non-root user setup
RUN groupadd --gid 1000 user && adduser --disabled-password --gecos '' --uid 1000 --gid 1000 user
WORKDIR /home/user
USER user

# Development stage setup
FROM base as dev
USER root
COPY --chown=user:user ./requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=user:user ./src /home/user/src
COPY --chown=user:user ./test /home/user/test
COPY --chown=user:user .env /home/user/.env
USER user
CMD ["python", "-m", "uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "5000", "--reload", "--env-file", "/home/user/.env"]

# Test stage setup
FROM base as test
COPY --chown=user:user ./requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
USER root
COPY --chown=user:user ./src /home/user/src
COPY --chown=user:user ./test /home/user/test
RUN mkdir /home/user/.mypy_cache && chown user:user -R /home/user/.mypy_cache
RUN pip install pytest
CMD ["pytest"]
USER user

# Default target stage
FROM dev

🤔 Current Challenge:
The error "Module magic not found" persists when running the project in a Docker container despite changing the library dependency. Seeking insights or suggestions to overcome this issue. Your thoughts? 👀

Regards 🚀

[Tianon Gravi]

If you apt-get install any Python program in this image, you will get a second copy of Python installed from the distribution. If you want to use our image and need modules like magic, you'll need to pip install, etc them instead.

[Charm AI]

Solution Implemented:
After encountering the "Module magic not found" error in the Docker container, I devised a solution by creating separate requirements.txt files for Windows and Linux versions of the project. The Linux version requires python-magic, while the Windows version necessitates python-magic-bin.

🐳 Dockerfile With Updated Configuration:

# Dockerfile for API service
FROM python:3.11.0 as base

# Define build arguments
ARG PROJECT=src
ARG TEST=test

# Install dependencies
RUN apt-get update && apt-get install -y \
    poppler-utils \
    tesseract-ocr \
    libgl1-mesa-glx \
    libmagic1 \
    python3-magic

# Create a non-root user
RUN groupadd --gid 1000 user && adduser --disabled-password --gecos '' --uid 1000 --gid 1000 user

WORKDIR /home/user
USER user

# Development stage setup
FROM base as dev

USER root
COPY --chown=user:user ./requirements-linux.txt ./
RUN pip install --no-cache-dir -r requirements-linux.txt
COPY --chown=user:user ./$PROJECT /home/user/$PROJECT
COPY --chown=user:user ./$TEST /home/user/$TEST
COPY --chown=user:user .env /home/user/.env
USER user
CMD ["python", "-m", "uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "5000", "--reload", "--env-file", "/home/user/.env"]

# Test stage setup
FROM base as test

COPY --chown=user:user ./requirements-linux.txt ./
RUN pip install --no-cache-dir -r requirements-linux.txt
COPY --chown=user:user ./$PROJECT /home/user/$PROJECT
COPY --chown=user:user ./$TEST /home/user/$TEST
RUN mkdir /home/user/.mypy_cache && chown user:user -R /home/user/.mypy_cache
RUN pip install pytest
CMD ["pytest"]
USER user

# Set the default target stage to 'dev'
FROM dev

🚀 Outcome:
By segregating the requirements between Windows and Linux versions and updating the Dockerfile correspondingly, the project now runs smoothly without the "Module magic not found" error in the Docker container.

[Tianon Gravi]

I will reiterate again that you do not want to apt-get install any python* APT packages with this Docker image. You either want to switch to using Debian directly (FROM debian:bookworm) and installing those from Debian, or switch to using "upstream" methods like pip for installing Python dependencies to keep using this image.