createIter can't be fed to a proc: zero_functional.nim(1675, 16) Error: internal error: proc has no result symbol
timotheecour opened this issue · comments
import zero_functional
import strutils
when defined(case1):
# error: /Users/timothee/.nimble/pkgs/zero_functional-#head/zero_functional.nim(1675, 16) Error: internal error: proc has no result symbol
# iterNode = quote:
proc bar[T](a:T)=
echo "in bar"
for si in a():
echo si
when defined(case2):
# error
proc bar[T](a:iterator: T {.inline.})=
echo "in bar"
for si in a():
echo si
when defined(case3):
# error
proc bar(a:iterator: string {.inline.})=
echo "in bar"
for si in a():
echo si
when defined(case4):
# works
template bar[T](a:T)=
echo "in bar"
for si in a():
echo si
proc test()=
lines(stdin) --> map(it.toUpper()) --> createIter(s)
# BUG
bar(s)
# ok
for si in s(): echo si
test()
@timotheecour
concerning proper error reporting: what version of nim do you use?
It does not look like a recent one, because toUpper(string) does not exist any more.
So maybe the nim documentation needs to be updated:
proc toUpper(s: string): string {..}
Converts s into upper case.
This works only for the letters A-Z. See unicode.toUpper for a version that works for any Unicode character.
Deprecated since version 0.15.0: use toUpperAscii instead.
Also: when posting a bug, next time: PLEASE try to call the "-->>" debug operator first - then look if the printed code has a bug. In 99% of the cases it has - trust me.
/cc @michael72
concerning proper error reporting: what version of nim do you use?
using latest devel:
nim --version
Nim Compiler Version 0.18.1 [MacOSX: amd64]
Compiled at 2018-08-01
Copyright (c) 2006-2018 by Andreas Rumpf
git hash: 8f4c5a8955594b4f2dc31f9629af9ad3d4780c2c
active boot switches: -d:release
It does not look like a recent one, because toUpper(string) does not exist any more.
So maybe the nim documentation needs to be updated
that's because you're reading docs from latest stable (https://nim-lang.org/docs/strutils.html) instead of latest devel; in devel, toUpper is not marked as deprecated and works. Would be nice to publish up to date docs though, feel free to upvote nim-lang/website#98
Also: when posting a bug, next time: PLEASE try to call the "-->>" debug operator first - then look if the printed code has a bug. In 99% of the cases it has - trust me.
Unfortunately I'm not sure how it applies in this case: you mentioned earlier "You can try using the -->> instead of --> then copy the output" however the bug occurs in bar(s) which doesn't have a --> ; in fact the line below works: for si in s(): echo si so teh pipeline lines(stdin) --> map(it.toUpper()) --> createIter(s) worked.
let me know if that's still unclear, thanks!
OK - my point is: is this a zero_functional bug? Or is it a nim bug? Or is it just a strange nim error mesage?
To me it looks like a nim bug - or maybe - yes - we need a closure iterator to pass around here. You haven't yet made your case though.
When writing lines(stdin) -->> map(it.toUpper()) --> createIter(s) you will get something like this in the compiler output:
iterator s(): auto =
for it0 in lines(stdin):
let it1 = it0.toUpperAscii()
yield it1use that code instead of the original line, then try to solve your problem with that code.
If you get working solution - e.g. by adding {.closure.} then we can try to add a {.closure.} pragma to the actual code generation - if that is possible.
indeed, adding {.closure.} AND resolving auto to string in iterator s(): auto makes it work:
import zero_functional
import strutils
when defined(case1):
# OK (used {.closure.} , and changed iterator s(): string to iterator s(): auto)
proc bar(a:iterator: string {.closure.})=
echo "in bar"
for si in a():
echo si
iterator s(): string {.closure.} =
for it0 in lines(stdin):
let it1 = it0.toUpper()
yield it1
bar(s)
when defined(case2):
# Error:
# Error: type mismatch: got <iterator (): string{.inline, gcsafe, locks: 0.}> but expected one of: proc bar(a: iterator (): string)
proc bar(a:iterator: string)=
echo "in bar"
for si in a():
echo si
iterator s(): string =
for it0 in lines(stdin):
let it1 = it0.toUpper()
yield it1
bar(s)OK - it looks like the code generation works with {.closure.} as well - unfortunately it is not feasible to detect the actual type of the iterator.
A possible solution would be to provide the type in the call, something like:
createIter(name,true,string) - that is createIter would take two optional arguments: closure (default is false) and resultType (default is auto).
unfortunately it is not feasible to detect the actual type of the iterator.
actually it's possible for inline iterators, not closure iterators, see:
- auto inference doesn't work with closure iterators: Error: expression 'myIter2()' has no type (inline are ok) nim-lang/Nim#8284
so a possible improvement would be to:
step 1: do auto type inference from the generated inline iterator:
iterator s(): auto =
for it0 in lines(stdin):
let it1 = it0.toUpperAscii()
yield it1
step 2: then use the auto resolved type to actually create the closure iterator, when the user asks for a closure iterator.
so user would just call: createIter(name, closure = true)
enhancement that could be done later
now, for closure iterators, since we're returning a closure (ie 1st class entity), we could have the following much cleaner syntax:
# old syntax
lines(stdin) --> map(it.toUpper()) --> createIter(myIter1, closure = true)
# new syntax (returns a closure iterator):
let myIter2 = lines(stdin) --> map(it.toUpper())
bar(myIter1) #ok
bar(myIter2) #okNo - no... really not.
The thing you really have to be clear about is - zero_functional is just a code generator. So basically it can only support what nim already supports. And there is no way in nim to simply state:
let myIter2 = iterator ....
We cannot support a different nim syntax just with macros. And as always - if you're unclear what is happening behind the curtains of zero_functional - just look at the generated code using the -->> debug operator.
OK - I probably found a way for using type deduction here - something like
proc printIt(a:iterator: int {.closure.}) =
for i in a():
echo($i)
proc test() =
iterator inlineIterator(): auto =
for i in 1..5:
yield i
iterator s(): type(inlineIterator()) {.closure.} =
for it in bla():
yield it
printIt(s)
test()And there is no way in nim to simply state: let myIter2 = iterator ....
there is, with closure iterators (but not inline iterators):
import strutils
template getIter1():untyped=
iterator s2(): string {.closure.} =
for it0 in lines(stdin):
let it1 = it0.toUpperAscii()
yield it1
s2
proc test()=
let myIter = getIter1()
for a in myIter(): echo a
test()And about type deduction: it really really is a messy thing in nim. Just try to give me a working example of your auto resolved type code for instance...
import strutils
import typetraits
import zero_functional
template getIter2():untyped=
lines(stdin) --> map(it.toUpper()) --> createIter(s)
type(s())
template getIter3(a:typed):untyped=
type(s())
template getIter4(a:untyped):untyped=
a --> createIter(s)
type(s())
proc test()=
block:
doAssert getIter2() is string
block:
doAssert getIter3(lines(stdin) --> map(it.toUpper()) --> createIter(s)) is string
block:
doAssert getIter4(lines(stdin) --> map(it.toUpper())) is string
test()maybe - but you have to think of --> as the entrance to the whole macro that generates the code and what is on the left side is not part of it, except the left hand argument.
So let x = [1,2,3] --> find(it == 2) will resolve in something like
let x = (proc (): auto =
for it0 in [1, 2, 3]:
if it0 == 2:
return some(it0)
else:
result = none(it0.type))()The point is: your code has to fit in all of the part on the right side of let x =
go figure
yes, that's what I have in mind.
let x = [1,2,3] --> find(it == 2)
# consume x (either via `for`, or via another proc that takes a closure iterator)
for ai in x(): echo aiwilll result in efficient code, probably identical in performance to inline iterator with compiler optimizations (depending on how complex the consuming function is), eg using link time optimization and compiler inlining.
And, we get benefits mentioned of composition (eg allowing to reuse code not built around zero-functional), and saner syntax.
OK - I've added the closure iterator first
as mentioned it can be created with createIter(<name>, true) or createIter(<name>, closure=true)
I will try to get the new syntax working next, but this will probably break existing code... I will do development on branch rework-iterator. Maybe first introduce also a compile time variable which has to be set when using the new feature. Problem is that old code created sequences, arrays or lists. With the new approach the to(seq) etc would have to be called explicitly. But I find this approach more consistent actually - the former approach could lead to surprising results.
awesome! can new syntax also work with inline iterators or just closure iterators?
I don't think so - first I will try closure iterator anyway. I guess inline iterator won't be possible - because as mentioned the left side of the assignment is not part of the --> macro and it is also not possible to return an inline iterator from a proc. Maybe you have an idea?
... there is also that: the closure iterator works with c but not with js backend!
[1,2,3] -->> map(it+1) --> createIter(s,true)
s() --> foreach(echo($it))If you look here on the nim iterators site you will find the statement:
...
- Closure iterators are not supported by the js backend.
tadaaa
I feel that additional support for closure iterators - as it is now - is OK, but I would not go any further as all other zero_functional features also work with js backend. I think that closure iterators should be fixed with the js backend first (if that can be fixed - well I think it should).
... but as a further step I would think about the to(iter) command - similar to to(seq) or to(list) etc.
This would actually return a closure iterator that would be incompatible with js, but maybe for js simply a seq could be returned.... Hmm... Well maybe that's the next step. And afterwards the default return type maybe could be set with a compiler option - thinking along your "case1"..."caseX" pattern with special compile switches...
Yes - so I will first implement the to(iter) which can be assigned to a variable. I hope that is possible, but it should.
well there are many things that are not supported by js backend, so code targetting js would anyway need to contain js specific conditional compilation. (plus there is compiling to wasm as another option, which could enable targetting js while avoiding js specific issues altogether -- untested though, need to try)
I think supporting closure iterators warrants sufficient benefits as discussed above (even if it means code targeting js will have to use "when" clauses)
Maybe you have an idea?
not at the moment; inline iterators are indeed limiting
... well both iterator types are
Btw I added an implementation that supports to(iter) - see also test.nim at the end of the file.
But there are some drawbacks: using the iterator inside another function again with zero_functional does not yield any results - because zero_functional creates an anonymous proc... see
nim-lang/Nim#8550
Anyhow - it works now , but has some limitations.
[edit: closure iterators work now in nested functions (e.g. the auto-function created by zero_functional) as well]
great, much cleaner now! just tried it, looks like these are now working
-
closureiterators creation - auto inference for closure iterators
- cleaner syntax with
let iter = ...instead ofcreateIterfor closure iterators
let it = lines(stdin) --> map(it.toUpper()) --> map(it.len) --> to(iter) - as a consequence,
itcan be fed into a proc that consumes a closure iterator
I wonder whether this could be made to work?
let it = lines(stdin) --> map(it.toUpper()) --> map(it.len) --> to(iter)
instead of:
let it = lines(stdin) --> map(it.toUpper()) --> map(it.len)
?
(ie, when at end of chain, produce closure iterator)
I wonder whether this could be made to work?
let it = lines(stdin) --> map(it.toUpper()) --> map(it.len) --> to(iter)
instead of:
let it = lines(stdin) --> map(it.toUpper()) --> map(it.len)
I think so - but it would break existing code as the current implementation tries to generate a collection of the input type - and if that is not possible - create a seq
But I agree that it might be more consistent. I'm still a little hesitent as iterators still have their little quirks here and there and they are not supported by JS. But that's not a reason to change the code. I'd probably go for a compile flag - at least for now.
I'd probably go for a compile flag - at least for now
+1
@michael72 maybe update your above post #36 (comment) (now that nim-lang/Nim#8550 was fixed) so we can keep track of any current issues?
eg using crossed out text
@timotheecour
Among a few other fixes and enhancements I created a new version 0.2.0 that also supports creating closures as default output type (in case when normally a sequence would be created) when the compile flag zf_iter is set.
https://github.com/zero-functional/zero-functional/tree/v0.2.0#compile-flags
So - I guess this can be closed now?