python module not found errors
Divyesh3011 opened this issue · comments
Facing Modules / package error
2023-07-12 17:08:15 Traceback (most recent call last):
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/decimal.py", line 3, in
2023-07-12 17:08:15 from _decimal import *
2023-07-12 17:08:15 ModuleNotFoundError: No module named 'numbers'
2023-07-12 17:08:15
2023-07-12 17:08:15 During handling of the above exception, another exception occurred:
2023-07-12 17:08:15
2023-07-12 17:08:15 Traceback (most recent call last):
2023-07-12 17:08:15 File "main.py", line 2, in
2023-07-12 17:08:15 from fastapi import FastAPI
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/site-packages/fastapi/init.py", line 7, in
2023-07-12 17:08:15 from .applications import FastAPI as FastAPI
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 3, in
2023-07-12 17:08:15 from fastapi import routing
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 20, in
2023-07-12 17:08:15 from fastapi import params
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/site-packages/fastapi/params.py", line 4, in
2023-07-12 17:08:15 from pydantic.fields import FieldInfo, Undefined
2023-07-12 17:08:15 File "pydantic/init.py", line 2, in init pydantic.init
2023-07-12 17:08:15 File "pydantic/dataclasses.py", line 3, in init pydantic.dataclasses
2023-07-12 17:08:15 import copy
2023-07-12 17:08:15 File "pydantic/class_validators.py", line 8, in init pydantic.class_validators
2023-07-12 17:08:15 File "pydantic/errors.py", line 1, in init pydantic.errors
2023-07-12 17:08:15 File "/usr/local/lib/python3.8/decimal.py", line 8, in
2023-07-12 17:08:15 from _pydecimal import *
2023-07-12 17:08:15 ModuleNotFoundError: No module named '_pydecimal'
Steps to Reproduce the Problem (DockerFile)
FROM python:3.8-slim-buster AS build
WORKDIR /app
COPY . .
RUN apt-get update && apt-get install -y --no-install-recommends
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir -r requirements.txt
Stage 2: Final stage
FROM python:3.8-slim-buster
WORKDIR /app
COPY --from=build /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
COPY . .
RUN pip install --no-cache-dir -r requirements.txt fastapi==0.68.1 uvicorn==0.15.0
CMD ["bash", "azure-soc2-manager.sh"]
EXPOSE 8000
Create preserved-paths.txt file in the final image
RUN echo "/usr/local/usr/local/lib/python3.8" > /preserved-paths.txt
- first i build the image using the above docker file
- Then i slim the image build
command use are
docker build -t soc2-test-1 . (to build-image)
docker-slim build --target soc2-test-11 --tag soc2-test-slim --http-probe=false
docker images | grep soc2
soc2-test-1 slim
cf086482e0b3 About a minute ago 33.9MB
soc2-test-1 latest
d3fdfab4467d 10 minutes ago 279MB
@Divyesh3011 what version of slim are you using? Also wonder why you disabled http probing: docker-slim build --target soc2-test-11 --tag soc2-test-slim --http-probe=false
Can you also describe what your app does and how the missing libraries are used in your app if you use them directly (e.g., pydantic)?
slim version linux|Transformer|1.40.2|762eea87bafc4b04fe6b0642a62534a148acdaca|2023-05-20_06:09:23AM
there is no specific reason behind disabling http probing i just found one of the eg similar to this so i used this cmd should i modify the cmd ?
the app is part of multiple microservices which are use to scan the image for any vulnerablity
this is my requirement file
i have multiple microservices most of them use the python:3.8-slim image as base & i am getting similar module/packages error
can you help me with a standard docker-file for python svc that would support docker-slim
requirement.txt
There's a couple of fastapi examples that might be useful for you: https://github.com/slimtoolkit/examples/tree/master/3rdparty/python_fastapi_official (and there's another one in the 3rdparty director, it has fastapi in its name too)
I thought you disabled http probing because you didn't have any EXPOSE instructions in your Dockerfile. You should have it, but if you don't you should use the --expose build command flag. Either of those ways tells slim the port to probe.
i tried updating my dockerfile according to your suggestion but getting this error i have added the slim-report file too
cmd : docker-slim build --target azure-soc2 --tag azure-soc2:slim
i have tried with this cmd too but getting the same error
docker-slim build --target azure-soc2 --tag azure-soc2:slim --expose=8000
dockerfile
FROM python:3.10-slim-buster
WORKDIR /app
COPY requirements.txt requirements.txt
RUN apt-get -y update
RUN pip3 install -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["bash", "azure-soc2.sh"]
cmd=build info=container message='obtained IP address' ip='172.17.0.2'
cmd=build info=cmd.startmonitor status='sent'
cmd=build info=event.startmonitor.done status='received'
cmd=build info=container name='slimk_27768_20230718143106' id='022295ba021c460c886551fbc3b3861caaee9f24cb9bc61b77b14f628423a5f1' target.port.list='49191' target.port.info='8000/tcp => 0.0.0.0:49191' message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER'
cmd=build state=http.probe.starting message="WAIT FOR HTTP PROBE TO FINISH"
cmd=build info=continue.after mode='probe' message='no input required, execution will resume when HTTP probing is completed'
cmd=build prompt='waiting for the HTTP probe to finish'
cmd=build state=http.probe.running
cmd=build info=http.probe.ports count='1' targets='49191'
cmd=build info=http.probe.commands count='1' commands='GET /'
cmd=build info=http.probe.call method='GET' target='http://127.0.0.1:49191/' attempt='1' error='Get "http://127.0.0.1:49191/": EOF' time='2023-07-18T14:31:20Z' status='error'
cmd=build info=http.probe.call error='Get "http://127.0.0.1:49191/": EOF' time='2023-07-18T14:31:36Z' status='error' method='GET' target='http://127.0.0.1:49191/' attempt='2'
cmd=build info=http.probe.call status='error' method='GET' target='http://127.0.0.1:49191/' attempt='3' error='Get "http://127.0.0.1:49191/": EOF' time='2023-07-18T14:31:52Z'
cmd=build info=http.probe.call attempt='4' error='Get "http://127.0.0.1:49191/": EOF' time='2023-07-18T14:32:08Z' status='error' method='GET' target='http://127.0.0.1:49191/'
cmd=build info=http.probe.call status='error' method='GET' target='http://127.0.0.1:49191/' attempt='5' error='Get "http://127.0.0.1:49191/": EOF' time='2023-07-18T14:32:24Z'
cmd=build info=http.probe.summary successful='0' total='5' failures='5'
cmd=build state=http.probe.done warning=no.successful.calls
cmd=build info=event message='HTTP probe is done'
cmd=build error=probe.error message='no.successful.calls'
slim: container stdout:
- Serving Flask app 'soc2' (lazy loading)
- Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead. - Debug mode: on
slim: container stderr:
WARNING:werkzeug: * Running on all addresses.
WARNING: This is a development server. Do not use it in a production deployment.
INFO:werkzeug: * Running on http://172.17.0.2:5000/ (Press CTRL+C to quit)
INFO:werkzeug: * Restarting with stat
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 139-218-447
slim: end of container logs =============
cmd=build state=exited code=-1
cmd=build info=container.inspector.cleanup name='slimk_27768_20230718143106' id='022295ba021c460c886551fbc3b3861caaee9f24cb9bc61b77b14f628423a5f1'
cmd=build state=container.target.shutdown.start
cmd=build state=container.target.shutdown.done
cmd=build info=report file='slim.report.json'
Can you describe what your app does?
the app is part of multiple microservices which are use to scan the image for any vulnerablity i
is there are way where i can specify the slim not to delete certain package like having a preserve.txt file where we keep all the required package & dependecy & tell slim to make sure all the package specified in the preserve.txt file are present in final Image
i have multiple microservice & for different svc i get different/package depedency error so having something like preserve.txt file will help to solve the errors
Docker file
FROM python:3.10-slim-buster
RUN pip3 install anyio==3.5.0
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["python", "start_server.py", "--reload","--port=8000"]
one of the depedecy-error i am facing (this dependecy is mentioned in requirement.txt file) with normal image it work but with slim image i get errors
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'anyio._backends'
@kcq
Traceback (most recent call last):
File "/app/start_server.py", line 1, in
from app.main import app
File "/app/app/main.py", line 17, in
from .models.discovery import ScanRQ,ScanRequest
ModuleNotFoundError: No module named 'app.models.discovery'
It shows no app.models.discovery but i have checked the path it contains the file
/app
└── app
├── init.py
├── main.py
└── models
├── init.py
└── discovery.py
docker-slim build --target gcr.io/***/aws-discovery-v2-dump:no-slim --tag gcr.io/***/aws-discovery-v2-dump:slim1 --http-probe=true --continue-after=40 --include-shell=true --include-path /usr/local/lib/