hitting google drive limits, torch/huggingface hub
AK391 opened this issue · comments
hitting drive limits on downloading models, another solution other than pydrive is weights could be under project release
see torch hub
https://pytorch.org/docs/stable/hub.html
"Pretrained weights can either be stored locally in the github repo, or loadable by torch.hub.load_state_dict_from_url(). If less than 2GB, it’s recommended to attach it to a project release and use the url from the release. In the example above torchvision.models.resnet.resnet18 handles pretrained, alternatively you can put the following logic in the entrypoint definition."
and a similar example from animegan hubconf, although the weights are much smaller in size here
https://github.com/bryandlee/animegan2-pytorch/blob/main/hubconf.py
or the models can be hosted on huggingface, see
Some files that need to be downloaded are too big to be uploaded to github and unfortunately git lfs is a paid service. From what I understand, huggingface hub only works for projects that are integrated with huggingface right? Like the transformer.
Not sure if there are other better free option other than gdrive to host the models.
@mchong6 huggingface hub supports models outside transformers, for example https://huggingface.co/sberbank-ai/rudalle-Malevich and https://huggingface.co/osanseviero/BigGAN-deep-128
Is the download limits still happening for you? For my models I dont mind uploading them, but I am not too sure about the pretrained models of other frameworks im using like stylegan2 and e4e.
@mchong6 was happening yesterday alot, not as much today. If you need help with uploading the models to huggingface hub, let me know, thanks
That would be really nice of you, I don't know much about huggingface 😅
@mchong6 ok ill look into it thanks
@mchong6 finetuned with 3 faces for 500 iters and added a model here https://huggingface.co/akhaliq/jojo-gan-spiderverse/blob/main/spiderverse-checkpoint-3-face-500iters.pt, overall the process was smooth, will try with other models
Thanks a lot 😊
@mchong6 its happening again, starting moving more models from google drive to huggingface hub
I appreciate it, thanks for doing this :)