Mistral Q8 Cuda invalid memory access
jafioti opened this issue · comments
Unclear why this occurs, seems to happen upon dropping a buffer in the copyfromdevice op, though it might just be that that was the last op to run before other buffers were dropped.
This only started happening after adding q8, didn't happen on llama which uses fp16