`GPTNeoX` incompatible with transformers >= 4.28.0
pcuenca opened this issue · comments
As discovered in #42.
The incompatibility was introduced in huggingface/transformers@7dcd870
Concretely, the reason for the problem lies in the use of torch.gather. When converted to Core ML, this assertion fails if shapes are flexible.
(There's a new implementation of gather_along_axis for iOS17 but by looking at the source code I don't think it would fix the problem).
The obvious workaround is to disable flexible shapes for GPTNeoX. This, in fact, is better for performance as flexible shapes don't seem to be compatible with GPU or ANE.
use_flexible_shapes is now False by default. Closing (wontfix) until we need to revisit.