salesforce / CodeGen

CodeGen is an open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Is there a way to prevent reloading parameters?

Penguin-jpg opened this issue · comments

Hello! I have been using CodeGen to generate lately. But I found that I spent most of the time on loading parameters.
I tried to separate out the create_model part to prevent it from reloading but cuda out of memory occurred.
So I want to know is there a way to prevent reloading parameters?
Thanks!

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

If you execute the last cell, it appears the amount of allocated cuda memory remains constant.

Does this help?

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

If you execute the last cell, it appears the amount of allocated cuda memory remains constant.

Does this help?

Thanks for reply!! It works 0.0.
I think I probably missed some lines from source code. Thanks again!

Glad it helps. Perhaps, you could update your shared notebook, so that users can easily try multiple samples?

Glad it helps. Perhaps, you could update your shared notebook, so that users can easily try multiple samples?

Updated!!

But I found that changing model will cause a TypeError.

I used 350M-mono to generate first and it works, then I changed to 2B-mono and reload the model. But when generating, it gave a TypeError.
image

Thanks for reporting this! I was able to reproduce the issue. This was because reloading of the tokenizer caused hiccups with max_model_input_sizes when tokenizing with truncation=True. I have modified the tokenizer call to explicitly use the specified max_length in this commit: 22b81d6.

Could you reset the runtime and try again?

There was a minor typo in the 350M-multi url. Here simplified code. Could you test and revise your shared notebook? Thank you.

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

@enijkamp @rooa Problems solved. Thanks for your help!! The notebook is updated.
Here is the link: https://colab.research.google.com/drive/1fQI8OgzMAR0bquCrvhlAtXSw6iMFbVgI?usp=sharing

Great, thanks!