Is there a plan to support 8bits (FP8 or INT8)?

[fromeast]

Is there a plan to support 8bits (FP8 or INT8)?

[Andrej]

There is plan for fp8, but not int8 - that is usually used in context of inference, this repo focuses on training right now. We will very likely get around to model inference as well and int8 will be relevant then, but that is still fairly far out, likely on the scale of few ~months. fp8 is more like scale of few ~weeks.

[Kozuch]

It is great to hear that FP8 in planned for training. Looks like there is lack of open source stack that deals with e2e LLM training process in the scale that an individual is able to comprehend and run. Thumbs up for this llm.c project!

Any updates on FP8 implementation? How difficult will be the code changes from current state?