Reproduction code runs super slow
BiEchi opened this issue · comments
I'm running vicuna-13b with config
---
prompt_file: prompts/asqa_default.json
eval_file: data/asqa_eval_gtr_top100.json
shot: 1
ndoc: 3
dataset_name: asqa
tag: gtr
model: alpaca-7b
temperature: 1.0
top_p: 0.95
and extra parameter --seed 43 --model vicuna-13b with offload=True. The reproduction is super slow as a result:
Loading checkpoint shards: 0%| | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards: 17%|█▋ | 1/6 [01:17<06:26, 77.24s/it]
Loading checkpoint shards: 33%|███▎ | 2/6 [02:52<05:51, 87.92s/it]
Loading checkpoint shards: 50%|█████ | 3/6 [04:45<04:57, 99.14s/it]
Loading checkpoint shards: 67%|██████▋ | 4/6 [06:34<03:26, 103.14s/it]
Loading checkpoint shards: 83%|████████▎ | 5/6 [08:26<01:46, 106.48s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [08:54<00:00, 79.83s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [08:54<00:00, 89.15s/it]
2023-06-06 16:30:27,946 - INFO - Finish loading in 537.02 sec.
2023-06-06 16:30:28,706 - INFO - Generating prompts...
0%| | 0/948 [00:00<?, ?it/s]
100%|██████████| 948/948 [00:00<00:00, 22713.88it/s]
2023-06-06 16:30:28,749 - INFO - Done.
0%| | 0/948 [00:00<?, ?it/s]2023-06-06 16:53:31,812 - INFO - Prompt length=1314
2023-06-06 16:53:31,812 - INFO - Question: Who has the highest goals in world football?
2023-06-06 16:53:31,812 - INFO - Gold answer: The players with the highest all-time goals and highest men's and women's international football goals differ. The player with the highest all-time men's football goals is Josef Bican, who in 2020 was recognized by FIFA, the international governing body of football, as the record scorer with an estimated 805 goals. Christine Sinclair has the highest goals in women's international football with 187 and is the all-time leader for international goals scored for men or women. Cristiano Ronaldo and Ali Daei are currently tied for leading goalscorer in the history of men's international football with 109.
2023-06-06 16:53:31,812 - INFO - Final model output: According to the IFFHS, Pelé is the player with the highest number of goals in world football, with a recognized total of 1281 goals [1]. However, the Soccer Statistic Foundation recognizes only 767 of Pelé's goals in official mode, making him the third highest scorer behind Josef Bican (805 goals) and Romario (772 goals) [1]. Maradona has been named the best soccer player in World Cup history by multiple sources, including The Times and FourFourTwo [2]. Additionally, Godfrey Chitalu of Zambia is claimed to have scored 116 goals (possibly 117) in 1972, which would beat Gerd Müller's record of 85 goals in a year, but FIFA has not recognized this achievement [2][3].
**0%| | 1/948 [23:03<363:49:19, 1383.06s/it]2023-06-06 17:11:41,101 - INFO - Prompt length=1308**
2023-06-06 17:11:41,101 - INFO - Question: Who is the original artist of sound of silence?
2023-06-06 17:11:41,101 - INFO - Gold answer: There are several songs with the title "Sound of Silence". Sounds of Silence is the second studio album by Simon & Garfunkel, released on January 17, 1966. The album's title is a slight modification of the title of the duo's first major hit, "The Sound of Silence", which was recorded in March 1964 and originally was released as "The Sounds of Silence". Another "Sound of Silence" is a song performed by Australian recording artist Dami Im, and is best known as Australia's entry at the Eurovision Song Contest 2016.
2023-06-06 17:11:41,101 - INFO - Final model output: "The Sound of Silence" is a song by the American music duo Simon & Garfunkel. It was written by Paul Simon and originally titled "The Sounds of Silence". The song was first recorded in 1964 and released on the duo's debut album "Wednesday Morning, 3 A.M." [1]. It was later re-released with electric guitar, bass and drums overdubs by Columbia Records staff producer Tom Wilson on the album "Sounds of Silence" [2]. The song has since been covered by various artists, including Disturbed, who recorded it for the video game "Rock Band 4" [3].
My machine is Tesla T4 with 16GB and it takes ~11GB to run the code. May I ask whether this speed is expected or there might be something wrong?
Hi,
It is possible that you are using cpu-offload because of insufficient GPU memory. In that case, inference will become extremely slow. I suggest you try turning off offload first, and then if there is GPU memory OOM, you might need to try a larger GPU.
Hi @gaotianyu1350,
May I ask if you have ever tried batch inference using either Vicuna or LLAMA in your work？
Best,
Yan
we haven't in this particular code base because batch implementation is very hard for the interactive/inlinesearch model, but for the vanilla method it is supposed to significantly speedup.
Just to offer more details, on our A100 GPU this code is able to finish Vanilla inference within 2 hours. So the 363h definitely looks off to me.
If I don't use offload_folder="offload" here: https://github.com/princeton-nlp/ALCE/blob/6f0e28a93dd8b6f25d3b0d48577594a32af2936f/utils.py#LL120C9-L120C27, the systems returns an error:
ValueError: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in this format.
Is it because our T5 is not able to run such large models?
Hi,
This error means that your GPU does not have enough memory to run this model. Maybe try a smaller LM or try a larger GPU?