Docker Installation Issue

Question

Docker Installation Issue

h0n33badg3rc0d3 opened this issue 10 months ago · comments

Following the docker install instructions here: https://hub.docker.com/repository/docker/laiyer/llm-guard-api/general
I run docker run -p 8000:8000 -d -e DEBUG='true' -e XFORMERS_MORE_DETAILS=1 llm-guard-docker:latest
The container exits and docker logs shows:

INFO:presidio-analyzer:Loaded recognizer: UsBankRecognizer
INFO:presidio-analyzer:Loaded recognizer: UsLicenseRecognizer
INFO:presidio-analyzer:Loaded recognizer: UsItinRecognizer
INFO:presidio-analyzer:Loaded recognizer: UsPassportRecognizer
INFO:presidio-analyzer:Loaded recognizer: UsSsnRecognizer
INFO:presidio-analyzer:Loaded recognizer: NhsRecognizer
INFO:presidio-analyzer:Loaded recognizer: SgFinRecognizer
INFO:presidio-analyzer:Loaded recognizer: AuAbnRecognizer
INFO:presidio-analyzer:Loaded recognizer: AuAcnRecognizer
INFO:presidio-analyzer:Loaded recognizer: AuTfnRecognizer
INFO:presidio-analyzer:Loaded recognizer: AuMedicareRecognizer
INFO:presidio-analyzer:Loaded recognizer: CreditCardRecognizer
INFO:presidio-analyzer:Loaded recognizer: CryptoRecognizer
INFO:presidio-analyzer:Loaded recognizer: DateRecognizer
INFO:presidio-analyzer:Loaded recognizer: EmailRecognizer
INFO:presidio-analyzer:Loaded recognizer: IbanRecognizer
INFO:presidio-analyzer:Loaded recognizer: IpRecognizer
INFO:presidio-analyzer:Loaded recognizer: MedicalLicenseRecognizer
INFO:presidio-analyzer:Loaded recognizer: SpacyRecognizer
INFO:presidio-analyzer:Loaded recognizer: PhoneRecognizer
INFO:presidio-analyzer:Loaded recognizer: UrlRecognizer
INFO:presidio-analyzer:Loaded recognizer: PatternRecognizer
INFO:presidio-analyzer:Loaded recognizer: PatternRecognizer
(…)ngual-nli-2mil7/resolve/main/config.json: 100%|██████████| 1.09k/1.09k [00:00<00:00, 2.25MB/s]
model.safetensors: 100%|██████████| 558M/558M [00:20<00:00, 27.4MB/s] 
(…)2mil7/resolve/main/tokenizer_config.json: 100%|██████████| 467/467 [00:00<00:00, 278kB/s]
spm.model: 100%|██████████| 4.31M/4.31M [00:00<00:00, 32.2MB/s]
tokenizer.json: 100%|██████████| 16.3M/16.3M [00:00<00:00, 26.8MB/s]
(…)nli-2mil7/resolve/main/added_tokens.json: 100%|██████████| 23.0/23.0 [00:00<00:00, 45.6kB/s]
(…)il7/resolve/main/special_tokens_map.json: 100%|██████████| 173/173 [00:00<00:00, 579kB/s]
(…)RTa-language-id/resolve/main/config.json: 100%|██████████| 756/756 [00:00<00:00, 1.20MB/s]
pytorch_model.bin: 100%|██████████| 336M/336M [00:12<00:00, 26.7MB/s] 
(…)ERTa-language-id/resolve/main/vocab.json: 100%|██████████| 994k/994k [00:00<00:00, 1.89MB/s]
(…)ERTa-language-id/resolve/main/merges.txt: 100%|██████████| 483k/483k [00:00<00:00, 2.95MB/s]
(…)ge-id/resolve/main/tokenizer_config.json: 100%|██████████| 19.0/19.0 [00:00<00:00, 20.6kB/s]
WARNING:xformers:WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.0 with CUDA None (you have 2.0.1)
    Python  3.10.13 (you have 3.10.13)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/xformers/_cpp_lib.py", line 121, in _register_extensions
    torch.ops.load_library(ext_specs.origin)
  File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 643, in load_library
    ctypes.CDLL(path)
  File "/usr/local/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/site-packages/xformers/_C.so: undefined symbol: _ZN3c106SymInt19promote_to_negativeEv

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/xformers/_cpp_lib.py", line 131, in <module>
    _build_metadata = _register_extensions()
  File "/usr/local/lib/python3.10/site-packages/xformers/_cpp_lib.py", line 123, in _register_extensions
    raise xFormersInvalidLibException(build_metadata) from exc
xformers._cpp_lib.xFormersInvalidLibException: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.0 with CUDA None (you have 2.0.1)
    Python  3.10.13 (you have 3.10.13)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
(…)-base-injection/resolve/main/config.json: 100%|██████████| 996/996 [00:00<00:00, 2.04MB/s]
pytorch_model.bin: 100%|██████████| 738M/738M [00:26<00:00, 27.9MB/s] 
(…)ction/resolve/main/tokenizer_config.json: 100%|██████████| 412/412 [00:00<00:00, 896kB/s]
spm.model: 100%|██████████| 2.46M/2.46M [00:00<00:00, 21.8MB/s]
(…)se-injection/resolve/main/tokenizer.json: 100%|██████████| 8.66M/8.66M [00:00<00:00, 15.3MB/s]
(…)injection/resolve/main/added_tokens.json: 100%|██████████| 23.0/23.0 [00:00<00:00, 37.9kB/s]
(…)ion/resolve/main/special_tokens_map.json: 100%|██████████| 173/173 [00:00<00:00, 289kB/s]
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     /home/user/nltk_data...
(…)c-comment-model/resolve/main/config.json: 100%|██████████| 704/704 [00:00<00:00, 1.28MB/s]
pytorch_model.bin: 100%|██████████| 268M/268M [00:10<00:00, 26.1MB/s] 
(…)model/resolve/main/tokenizer_config.json: 100%|██████████| 403/403 [00:00<00:00, 2.28MB/s]
(…)xic-comment-model/resolve/main/vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 5.93MB/s]
(…)omment-model/resolve/main/tokenizer.json: 100%|██████████| 466k/466k [00:00<00:00, 10.9MB/s]
(…)del/resolve/main/special_tokens_map.json: 100%|██████████| 112/112 [00:00<00:00, 457kB/s]
(…)model/resolve/main/tokenizer_config.json: 100%|██████████| 2.00/2.00 [00:00<00:00, 6.57kB/s]
(…)detection-model/resolve/main/config.json: 100%|██████████| 657/657 [00:00<00:00, 1.20MB/s]
(…)s-detection-model/resolve/main/vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 12.2MB/s]
(…)del/resolve/main/special_tokens_map.json: 100%|██████████| 112/112 [00:00<00:00, 215kB/s]
tf_model.h5: 100%|██████████| 268M/268M [00:14<00:00, 18.3MB/s] 
2023-10-16 16:13:03.486775: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
2023-10-16 16:13:03.646938: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
2023-10-16 16:13:03.701131: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
2023-10-16 16:13:03.945515: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 9437184 exceeds 10% of free system memory.
2023-10-16 16:13:03.953374: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 9437184 exceeds 10% of free system memory.
(…)ware-url-detect/resolve/main/config.json: 100%|██████████| 1.01k/1.01k [00:00<00:00, 5.52MB/s]
pytorch_model.bin: 100%|██████████| 711M/711M [00:21<00:00, 32.4MB/s] 
(…)etect/resolve/main/tokenizer_config.json: 100%|██████████| 367/367 [00:00<00:00, 1.20MB/s]
(…)alware-url-detect/resolve/main/vocab.txt: 100%|██████████| 996k/996k [00:00<00:00, 30.5MB/s]
(…)ect/resolve/main/special_tokens_map.json: 100%|██████████| 125/125 [00:00<00:00, 510kB/s]
(…)li_R1_R2_R3-nli/resolve/main/config.json: 100%|██████████| 703/703 [00:00<00:00, 552kB/s]
pytorch_model.bin: 100%|██████████| 1.43G/1.43G [00:49<00:00, 29.1MB/s]

I see there's an error with xformers. How do I fix this?
Running on mac os m1 ventura 13.6

Oleksandr Yaremchuk · Answer 1 · Tue Oct 17 2023 05:02:12 GMT+0800 (China Standard Time)

Hey @h0n33badg3rc0d3 ,
Thanks for submitting the issue.

30 minutes ago, we released the new version (v0.3.0) to the docker. It uses Python 3.11. I checked on my Macbook Pro M1Pro today.

Can you please try again to pull the new image and run it?

h0n33badg3r · Answer 2 · Tue Oct 17 2023 08:12:23 GMT+0800 (China Standard Time)

Latest push doesn't seem to help. Interestingly, even when i delete the prior image and then re-pull, i get the same image I had 11 hours ago. The hash of the latest image I pull matches the hash on hub.docker too-- 63d4a3a440e7f3335da507c03a76a8da2202d1d1a9605c1ae8a5100d68e334c1

On a different note, are you able to create an image for linux/amd64 architecture too?

Oleksandr Yaremchuk · Answer 3 · Tue Oct 17 2023 19:13:26 GMT+0800 (China Standard Time)

I used this command on my Mac

docker run -p 8000:8000 -d -e DEBUG='true' -e XFORMERS_MORE_DETAILS=1 laiyer/llm-guard-api:latest

And got this response

Digest: sha256:63d4a3a440e7f3335da507c03a76a8da2202d1d1a9605c1ae8a5100d68e334c1
Status: Downloaded newer image for laiyer/llm-guard-api:latest
d6fe92d1e3a7e336c1ea3b37456656343d5753db31ec04b0453b31513880b064

Then launched the container and after some time of loading all the models, it worked.

On the other question, I put it on the list for the next version. Thanks for the suggestion.

Oleksandr Yaremchuk · Answer 4 · Sun Oct 29 2023 15:09:05 GMT+0800 (China Standard Time)

Hey @h0n33badg3rc0d3 , I added images for 2 types of architectures https://hub.docker.com/repository/docker/laiyer/llm-guard-api/tags?page=1&ordering=last_updated