Clarification on use of AutoGPTQ, etc.
uogbuji opened this issue · comments
Hi, I noticed the removal of AutoGPTQ build in the latest. Is it because of this bug?AutoGPTQ/AutoGPTQ#128
Sounds like @PanQiWei is on it, so that's good, but my biggest question is whether this effectively removes the GPU support. I admit I'm still a bit new to this space, so please let me know if there is some alternative mechanism in place now for GPU support. I had been downloading .safetensors versions of models, but should I be looking for something different now?
@uogbuji Thank you for flagging that bug in AutoGPTQ. The separate install directives were removed from the Dockerfile because AutoGPTQ is now included by default in the upstream repository requirements (and should be picked up during the core installation process). There is certainly no intention for regression / removal and I definitely want that library included in all images.
I have tested the current docker hub image and AutoGPTQ is functional on my inference machine (PopOS 20.04, RTX 3090) so I do not think this repo is currently affected. Would you mind testing it on your machine to see if that is also the case? Or - are you trying to build the image and running into a problem? Either way, it's good to know because it's an important feature.
I will leave this issue open for the time being. Input and experiences from other users is also welcomed!
AutoGPTQ is functional in the latest version and was also functional in the previous version so I am going to close this issue. Thank you again for bringing the upstream repo's docker build issue to my attention, and please open a new issue if you have any further problems or suggestions.