pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Downloads the whole hf repo

das-projects opened this issue · comments

The scripts/download.py downloads the whole hf repo including the *.safetensors files, even though it seems only the *.bin files are required.

It usually a bad practice for repo creator to make multiple copies of the same file just different serialization format, like .bin and .safetensor.

usually, hf transformer perfers .safetensor format becuase loading .bin could lead to executing harmful code if .bin is hacked.

The hf transformer provides python inferface for downloading specific formats from hf repo, but I don't think this project use that interface

snapshot_download(repo_id="", ignore_patterns=["*. safetensors"])

will do