[ENHANCEMENT] Automatically use 'best' available device when none (or an invalid one) is specified
OwenPendrighElliott opened this issue · comments
Is your feature request related to a problem? Please describe.
When switching Marqo between devices you currently have to be sure you update every piece of code that specifies a device. This is most problematic in the cloud where nodes can be swapped from CPU to GPU and vice versa easily.
Describe the solution you'd like
Similar to how HuggingFace pipelines automatically use a GPU if one is available I'd like Marqo to automatically use the best available device if one is not specified. Additionally, if one is specified but is not available then it should give a warning saying it has defaulted to whichever device was available.
Example Scenarios:
- I have a GPU and don't tell Marqo what device I want (either in the cloud or using a container run with
--gpus all)- The GPU is detected and used instead of CPU
- My system only has a CPU but I ask Marqo to use a GPU
- Marqo logs a warning (to the docker console and python client) like
Warning: request received with "device='cuda'" however no CUDA compute capable devices were found, Marqo will default to CPU inference.
- Marqo logs a warning (to the docker console and python client) like
Describe alternatives you've considered
While I think both scenarios should be catered for, the first one is the most important.
Additional context
If you never specify a device in your code then you should be able to swap between CPU and GPU Marqo setups without having to make any code changes while also getting to use the hardware acceleration when it is available.
