KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'
WindyOrgUK opened this issue · comments
Not sure which package/version causes the problem as I had all working perfectly before on Ubuntu 20.04 with RTX 3090 GPU.
I then tried to reinstall localGPT from scratch and now keep getting the following for GPTQ models.
Exactly the same error when I just tried the freshly built docker:
2024-01-11 19:23:27,421 - INFO - SentenceTransformer.py:66 - Load pretrained SentenceTransformer: hkunlp/instructor-large
...
2024-01-11 19:43:22,290 - INFO - run_localGPT.py:59 - Loading Model: TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ, on: cuda
2024-01-11 19:43:22,290 - INFO - run_localGPT.py:60 - This action can take a few minutes!
2024-01-11 19:43:22,290 - INFO - load_models.py:94 - Using AutoGPTQForCausalLM for quantized models
and when the system runs and I send my first prompt, I get:
...
...
File "/usr/local/lib/python3.10/dist-packages/transformers/cache_utils.py", line 78, in getitem
raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'
I did further tests.
- All python version uninstalled and all local python packages deleted and all pip caches deleted.
- New user created (so no issues with user's .cache folder possile)
- Python 3.10 installed from scratch
- localGPT installed from scratch
- First start with default constatnts.py using llama-cpp based model - all works OK
- First start with constatnts.py using GPTQ model - same eror:
File "/home/ai/.local/lib/python3.10/site-packages/transformers/cache_utils.py", line 78, in getitem
raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'
Encountering the same issue, have you got the fix for this?
I am facing this as well. Something to do with transformers library. If anyone solved it please share the solution. I will try to rollback to previous version for now.