Runtime GPU backend selection

Question

Runtime GPU backend selection

philpax opened this issue a year ago · comments

At present, you can only compile support for one backend at a time, and you're locked into the choice of backend. As CLBlast and CuBLAS are both supported on Linux and Windows, this means that developers must distribute twice the number of binaries, and users may not be sure which binary to download.

I believe it may be possible to build support for all backends for a given platform, and switch which one gets activated at runtime. This option would only be on start-up (i.e. the application would need to restart to switch backends, due to ggml's leaky abstractions), but I think it should be possible otherwise.

This would allow developers to distribute one binary per platform, and (potentially) automatically select the best backend based on the available hardware.

Clark McCauley · Answer 1 · Mon Jul 24 2023 01:15:42 GMT+0800 (China Standard Time)

Hilarious, I'm literally halfway through writing up a question on your GPU documentation PR about this. I would be interested in this for my little app project. Ideally I wouldn't have to release a binary for each OS/backend combination.

Philpax · Answer 2 · Mon Jul 24 2023 01:30:08 GMT+0800 (China Standard Time)

Yeah, don't worry, I'm thinking about downstream deployment concerns too 😅

Lukas Kreussel · Answer 3 · Mon Jul 24 2023 02:19:14 GMT+0800 (China Standard Time)

Shouldn't switching the backend be possible after ggerganov/llama.cpp#2239 is implemented?

Philpax · Answer 4 · Mon Jul 24 2023 02:50:47 GMT+0800 (China Standard Time)

Not sure if that enables runtime switching. Should definitely ease the implementation of this (once we eat the initial pain of porting over), but I think it's still possible to do with the current interface (...assuming none of the backends initialize magic static state prior to any functions being called)