WARNING: The NVIDIA Driver was not detected
savikko opened this issue · comments
When launching to fly, this pops up on the logs.
Is it still using the GPU or is there something wrong on the config?
2024-02-23T14:46:56.405 app[e2865d1efe3578] ams [info] https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2024-02-23T14:46:56.425 app[e2865d1efe3578] ams [info] WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
2024-02-23T14:46:56.425 app[e2865d1efe3578] ams [info] Use the NVIDIA Container Toolkit to start this container with GPU support; see
2024-02-23T14:46:56.425 app[e2865d1efe3578] ams [info] https://docs.nvidia.com/datacenter/cloud-native/ .
This might be just timing error:
https://fly.io/blog/transcribing-on-fly-gpu-machines/
"Note: As my machine started up, I saw a log line WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available., which ended up being an issue of timing. Once everything is running, I was able to see things were working by using fly ssh console -a $APP_NAME and running command nvidia-smi to confirm that the VM had a GPU. It also listed the running web service (Python in this case) was running as a GPU process."
Yup you're right, it's a timing issue between the startup time where no driver is detected to the model being loaded. Wouldn't affect your deployment :)