Giters

leondz / garak

LLM vulnerability scanner

Home Page:https://discord.gg/uVch4puUCs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CUDA out of memory

kbmlcoding opened this issue · comments

commented

Running below command on Centos 7 with 2 32GB Nvidia-tesla v100 gpu card. model "microsoft/Phi-3-mini-4k-instruct" and probe "dan"
I have enough memory 32 GB to hold this ~3B llm but not sure why i am getting below error..
command:

python3 -m garak --model_type huggingface --model_name microsoft/Phi-3-mini-4k-instruct --report_prefix microsoft_Phi-3-mini-4k-instruct --probes dan
error:

024-06-07 14:05:59,485  DEBUG  harness: probe start for garak.probes.dan.Dan_10_0
2024-06-07 14:05:59,485  DEBUG  probe execute: <garak.probes.dan.Dan_10_0 object at 0x7f6c0087dfc0>
2024-06-07 14:06:04,066  ERROR  CUDA out of memory. Tried to allocate 88.00 MiB. GPU 
2024-06-07 14:06:04,067  DEBUG  probe return: <garak.probes.dan.Dan_10_0 object at 0x7f6c0087dfc0> with 1 attempts

stdout:

dan.Dan_10_0                                                                                 dan.DAN: SKIP  ok on    0/   0                              
dan.Dan_10_0                                                             mitigation.MitigationBypass: SKIP  ok on    0/   0
commented

closing for now .. after enabling multi-gpu the error is gone !