`FiberError` with `Enumerator` on Ruby < 3.0

[George Claghorn]

I’m consistently getting a FiberError back from magnus::Enumerator::next on Ruby 2.6 and Ruby 2.7:

#<FiberError: fiber called across stack rewinding barrier>

See this serde-magnus CI run for an example.

Strange details:

• Only on x86-64 Linux. I haven't been able to reproduce the issue on an Apple Silicon Mac.
• Only for heterogeneous arrays. [1234, true, "Hello, world!"], not [123, 456, 789].

I suspect ruby/ruby#4606 fixed this in Ruby 3.0. The timing isn’t right. 🤷‍♂️

Not sure if there’s anything to be done about this. Wanted to flag just in case.

[Mat Sadler]

Huh, weird, I've seen that error before, but the other way round, when calling Ruby's Enumerator#next on an enumerator produced from a method implemented with Magnus.

So given this pattern:

def example
  return to_enum(:example) unless block_given?
  yield 1
  yield 2
  yield 3
  nil
end

example {|i| p i}   # prints "1\n2\n3\n"
e = example         #=> #<Enumerator: main:example>
e.next              #=> 1
e.next              #=> 2

You'd expect to be able to write the following with Magnus:

fn example(rb_self: Value) -> Result<Value, Error> {
    if !block::block_given() {
        return Ok(*rb_self.enumeratorize("example", ()));
    }
    let _: Value = block::yield_value(1)?;
    let _: Value = block::yield_value(2)?;
    let _: Value = block::yield_value(3)?;
    Ok(*QNIL)
}

However that gets you #<FiberError: fiber called across stack rewinding barrier> when you call #next.

---

Aside: Now I try this, the error is only occurring on Ruby <= 3.0. I'm pretty sure 3.0 was the latest release when I first encountered this. I've only tested this particular case on M1 (originally) and M2 (now) Macs.

---

I tracked this down to the call to rb_protect that Magnus puts around all calls to Ruby to catch exceptions. So for that case I developed a workaround (of all the code in Magnus this is the bit I'm least confident is correct) so you can instead write:

fn example(rb_self: Value) -> block::Yield<impl Iterator<Item = i64>> {
    if block::block_given() {
        block::Yield::Iter((1..=3).into_iter())
    } else {
        block::Yield::Enumerator(rb_self.enumeratorize("example", ()))
    }
}

and have it work correctly with either a block or returning an Enumerator.

I don't think I can safely do the same thing for <Enumerator as Iterator>::next(), as its called by end user code where it wouldn't be safe to let a Ruby exception propagate through Rust code.
Edit: Oh, yeah, I really can't use that approach because Ruby Enumerators signal they are finished by raising an exception.

[Mat Sadler]

I adapted an existing test to use a heterogeneous array and can't reproduce this on CI main...heterogeneous-enumerator-test