how to use all gpus ?
ucas010 opened this issue · comments
hi,dear
I want to use all GPUs,but only the first to be used when inference
docker --version
Docker version 23.0.1, build a5ee5b1
cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
run code:
docker run --gpus all -p 8501:8501 --name what --mount type=bind,source=/data...l/,target=/models/../ -e MODEL_NAME=what -t tensorflow/serving:latest-gpu &
others bugs
unknown flag: --max_batch_size
See 'docker run --help'.
unknown flag: --enable_batching
See 'docker run --help'.
unknown flag: --num_batch_threads
See 'docker run --help'.
Following nvidia docker user guide, using all or comma separated list of GPU indexes
for example --gpus '"device=1,2"' should utilize all the gpus.
Can you try adding --privileged=true in docker run command. This gives access to all devices in the machine (including all GPUs) regardless of your —gpus setting. Let us know if this helps.
Thank you!
not useful ,
TF Serving only executes the graph loaded. If you graph is using multiple GPUs TF serving will use them, if you graph is only using one GPU (most common case) then you will be able to run it in one GPU in TF Serving.
Probably your best solution is to build a script which loads your graph once per GPU, then uses some code in the CPU to split the batch data into the GPU graphs and finally export the whole graph with support for multiGPU.
Similar feature request #311 and #1588 were also filled in past but couldn't make it because of not enough community support. Kindly let us know if the above approach works for you, else we can consider this feature request.
Thank you!
This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.
This issue was closed due to lack of activity after being marked stale for past 7 days.
