ValueError: cannot create std::vector larger than max_size()

[jian hu]

I am very grateful to ByteDance for making such a helpful product!

But I encountered some problem in the using. I want to do int8 inference on GPT2(13B) using lightseq.

here is my question:

step 1：
python export/huggingface/hf_gpt2_export.py
transfer gpt2(13B) to hdf5
and "lightseq_gpt2_large.hdf5" need 96GB disk to storage.

step 2：

>>> import lightseq.inference as lsi
>>>model = lsi.Gpt("lightseq_gpt2_large.hdf5", max_batch_size=12)

Parsing hdf5: lightseq_gpt2_large.hdf5
loading 1131 MB of embedding weight.
finish initializing emb_wei from host to device
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot create std::vector larger than max_size()

System message:
GPU:A10 (24GB)
mem: 80GB
disk: 500GB
cuda:11.3
torch:10.1
tranformers:4.26.1

and i have another question.
what's the difference between these four files?
export/huggingface/hf_gpt2_export.py
export/huggingface/ls_hf_gpt2_export.py?
export/huggingface/ls_hf_quant_gpt2_export.py?
export/huggingface/ls_torch_hf_quant_gpt2_export.py?

Thank!

[jian hu]

I found the answer

[Chen Zhengda]

I found the answer

I encountered the same error in other cases. How specifically was it resolved?