Giters

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models

Home Page:https://docs.rs/llm/latest/llm/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Free tensors from RAM if they are offloaded to an Accelerator

LLukas22 opened this issue · comments

commented

Right now the data of a tensor isn't freed if it is offloaded to a GPU. We should fix that to enable users to run bigger models which are split between CPU and GPU.