Run w/ CUDA GPU?

[Shane Neeley]

This is all about CPU workers. What about GPU? Is there some way to tell uvicorn / gunicorn to run the worker on GPU?

Using nvidia-docker instead of docker, it should be possible. But I think I need to tell Gunicorn to boot workers on the GPU devices?

[Sebastián Ramírez]

Hmm, not really, everything runs on the CPU and only very specific custom and complex code would run on GPU.

To make something run on GPU you would use something like a deep learning framework like PyTorch or a tool to perform Numpy-like operations in GPU, like https://cupy.chainer.org/. But it would be only for Numpy-like operations. Linear algebra stuff in big matrices and vectors, that's what GPUs can do very well.

But the rest of the API code would still run in CPU.

Now, to use nvidia-docker you might want to look at https://github.com/tiangolo/uvicorn-gunicorn-machine-learning-docker

I haven't been able to update it in a while but might still help you a bit.

[github-actions]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

[Shane Neeley]

Thanks for the response. I eventually got a system working with nvidia-docker and flask. I wasn't aware of that ML repo you made. I'll check that out for my next project.

[Juan Acevedo]

If anyone needs this, I did it as follows:

FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu18.04

RUN apt-get update
RUN apt-get install software-properties-common -y
RUN add-apt-repository ppa:deadsnakes/ppa -y
RUN apt-get install python3.8 -y
RUN apt-get install python3.8-distutils -y

RUN apt-get install python-dev python3-dev python3.8-dev \
     build-essential libssl-dev libffi-dev \
     libxml2-dev libxslt1-dev zlib1g-dev -y

RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc

RUN apt-get install curl wget -y

RUN apt-get install linux-libc-dev -y

RUN apt-get install libc6-dev -y

RUN wget https://bootstrap.pypa.io/get-pip.py
RUN  python3.8 get-pip.py

ENV APP_HOME=/app

WORKDIR $APP_HOME

COPY . /app

RUN pip3.8 install --no-cache-dir -r requirements.txt

EXPOSE 80

CMD ["gunicorn","main:app","-w","1","--threads", "1", "--timeout", "1500", \
  "--worker-connections=300","-k","uvicorn.workers.UvicornWorker","--bind", "0.0.0.0:80"]

My requirements.txt:

tensorflow
tensorflow-hub
matplotlib
six
pillow
notebook
pydantic
fastapi
uvicorn[standard]
gunicorn

Probably don't need the notebook there. Also the big timeout time is due to me loading this in my laptop that doesn't have cuda installed so the model takes forever to load and I need to increase the timeout for testing. on a GPU you wouldn't need it.