Bug: External iterators in nested `for` loops

[Izaakwltn]

With nested for loop, external/ parametric iterators cause unexpected behavior.

Defining iterators within each for macro gives expected results

(coalton-toplevel

  (define (permutations a-limit b-limit c-limit d-limit)
    (let results = (cell:new Nil))
    (for a in (iter:up-to a-limit)
         (for b in (iter:up-to b-limit)
              (for c in (iter:up-to c-limit)
                   (for d in (iter:up-to d-limit)
                        (cell:push! results (Tuple4 a b c d))))))
    (reverse (cell:read results))))

NESTED-LOOP-BUG> (coalton (permutations 2 2 2 2))
(#.(TUPLE4 0 0 0 0) #.(TUPLE4 0 0 0 1) #.(TUPLE4 0 0 1 0) #.(TUPLE4 0 0 1 1)
 #.(TUPLE4 0 1 0 0) #.(TUPLE4 0 1 0 1) #.(TUPLE4 0 1 1 0) #.(TUPLE4 0 1 1 1)
 #.(TUPLE4 1 0 0 0) #.(TUPLE4 1 0 0 1) #.(TUPLE4 1 0 1 0) #.(TUPLE4 1 0 1 1)
 #.(TUPLE4 1 1 0 0) #.(TUPLE4 1 1 0 1) #.(TUPLE4 1 1 1 0) #.(TUPLE4 1 1 1 1))

---

Defining the iterator externally and calling it as an argument causes unexpected results, quitting prematurely:

(coalton-toplevel

  (define (permutations2-backend a-iter b-iter c-iter d-iter)
    (let results = (cell:new Nil))
    (for a in a-iter
         (for b in b-iter
              (for c in c-iter
                   (for d in d-iter
                        (cell:push! results (Tuple4 a b c d))))))
    (reverse (cell:read results)))

  (define (permutations2 a-limit b-limit c-limit d-limit)
    (permutations2-backend (iter:up-to a-limit)
                           (iter:up-to b-limit)
                           (iter:up-to c-limit)
                           (iter:up-to d-limit))))

NESTED-LOOP-BUG> (coalton (permutations2 2 2 2 2))
(#.(TUPLE4 0 0 0 0) #.(TUPLE4 0 0 0 1))

---

It seems to work okay for the innermost level, and this isn't a problem for non-nested for loops, for example:

(coalton-toplevel

  (define (single-for-taking-iter it)
    (let results = (cell:new Nil))
    (for i in it
         (cell:push! results i))
    (reverse (cell:read results))))

NESTED-LOOP-BUG> (coalton (single-for-taking-iter (iter:up-to 7)))

(0 1 2 3 4 5 6)

---

(Package Definition):

(defpackage #:nested-loop-bug
  (:use #:coalton
        #:coalton-prelude)
  (:local-nicknames (#:tuple #:coalton-library/tuple)
                    (#:cell #:coalton-library/cell)
                    (#:iter #:coalton-library/iterator)))

[Izaakwltn]

By the way, this is also the case for iter:for-each!

[Colin OKeefe]

I don't think this is a bug -- the issue is that the same iterator object cannot be iterated over twice.

When you call (permutations2-backend iter-a iter-b iter-c iter-d) each of those iterator arguments can only be iterated over once. You'll notice that in this example, iter-d is consumed after it produces the value 1, after that it can no longer be used. Hence there's nothing left for the loop to do, so you only get those first two values because the push operation is never called again.

But in the first example, where you are creating a fresh iterator object using the value most recently generated in the nearest outer-loop, it works as expected.

[Colin OKeefe]

There IS an inconsistency here though. If you replace the iterator objects with objects of types that are merely into-iterator but not iterator, you get the expected result:

(coalton 
       (permutations2-backend "ab" "cd" "ef" "gh"))
(#.(TUPLE4 #\a #\c #\e #\g) #.(TUPLE4 #\a #\c #\e #\h)
 #.(TUPLE4 #\a #\c #\f #\g) #.(TUPLE4 #\a #\c #\f #\h)
 #.(TUPLE4 #\a #\d #\e #\g) #.(TUPLE4 #\a #\d #\e #\h)
 #.(TUPLE4 #\a #\d #\f #\g) #.(TUPLE4 #\a #\d #\f #\h)
 #.(TUPLE4 #\b #\c #\e #\g) #.(TUPLE4 #\b #\c #\e #\h)
 #.(TUPLE4 #\b #\c #\f #\g) #.(TUPLE4 #\b #\c #\f #\h)
 #.(TUPLE4 #\b #\d #\e #\g) #.(TUPLE4 #\b #\d #\e #\h)
 #.(TUPLE4 #\b #\d #\f #\g) #.(TUPLE4 #\b #\d #\f #\h))

So there is a disappointing inconsistency. When "ab" is turned into an iterator, a fresh iterator is made. But when the value of (iter:up-to 10) is passed to into-iterator's method, you get the exact same object back again.

[Colin OKeefe]

For a more instructive example:

  (define (run-it) 
      (let ((x (iter:up-to 3)))
        (iter:for-each! (fn (i) (traceobject "hey" i)) x)
        (iter:for-each! (fn (i) (traceobject "nope" i)) x)))

This will print out

(coalton (run-it ))
hey: 0
hey: 1
hey: 2
COALTON::UNIT/UNIT

[Izaakwltn]

Thank you for the illuminating response! I forgot that iterators were single-use...

I'll close the issue

[Izaakwltn]

I'm closing the issue

Edit: It won't let me close the issue.

[Colin OKeefe]

To be clear, I think that there is an "issue" here, just not a bug.

I think it is genuinely unsettling from a user perspective that, because for uses into-iter under the hood, we observe inconsistent behavior between non-ITERATOR instances of INTOITERATOR and ITERATORS