expandMacros results in illformed AST

[Vindaar]

Just stumbled on this while (finally!) trying the library. Replaced a chain of sequtils procs with zero_functional equivalents and everything worked. Then I was interested in the code it produces, so I used expandMacros. However, that results in an

~/.nimble/pkgs/zero_functional-0.0.7/zero_functional.nim(271, 6) Error: illformed AST: proc (): auto =

The simple code I ran:

import macros
import strutils
import zero_functional

const filename = "half_life_muon.txt"

expandMacros:
  let s = readFile(filename).splitLines --> filter('#' notin it and it.len > 0).
                                            map(it.splitWhitespace)

where the file is just

# bin   counts
1       1477
2       865
3       533
4       353
5       207
6       133
7       100
8       60
9       38
10      3

I'm running Nim devel from yesterday nim-lang/Nim@0da8793.

Maybe I'm missing something, but just wanted to report it now, because otherwise I'll forget about it again. :)

[Michael Schulte]

@Vindaar
Thanks for trying it
Well "simple" is probably a point of view thing...
I am not sure what you are actually trying here. Also looks like we need to update to a new version with nimble - I'll do that later. In the meanwhile you could also simply download the latest version of zero_functional.nim and give it a try.
Anyway: we have an iterator that returns string (splitLines) and another such iterator (splitWhitespace)
What you are actually getting is an iterator of iterator of string.
What you might want to get is an iterator of strings - right? So you should probably use the flatten function.

Automatic determination of the output type is sometimes difficult - especially when working with iterators as input type (this is more of a shortcoming of nim).
You can specify an explicit output type using .to(seq[string]) for instance and probably get better error messages then.
Setting another iterator here can also be done using createIter(s) as last function call.

Try for instance:

readFile(fileName).splitLines --> filter(...) --> map(...) --> flatten() --> createIter(s)

or

let s = readFile(fileName).splitLines --> filter(...) --> map(...) --> flatten() --> to(seq[string])

or without flatten

let s = readFile(fileName).splitLines --> filter(...) --> map(...) --> to(seq[seq[string]])

not sure what you are actually trying here though...

[Vindaar]

Maybe I'm just doing weird things after all or misunderstanding what I'm doing, haha.

My idea is:

• readFile -> string of file
• splitLines -> seq[string], lines of file
• filter -> seq[string], all lines without header & non empty
• map -> seq[seq[string]] -> split two columns into seq[string] each
  edit: should clarify that last statement: split each line into a seq[string], so that s contains all lines of the file, where each line is a seq[string], with element 0 being the first columns' value and 1 the second.

I'm a little confused by your statement regarding splitLines and splitWhitespace. I'm calling
https://nim-lang.org/docs/strutils.html#splitWhitespace,string,int
and splitLines below that as a proc and not as an iterator? Unless Nim instead uses the iterator here, which I don't even want. Or does zero-functional take the iterators explicitly?

I've updated to the current head and the problem still persists. Also tried your suggestions: for the createIter case, I get a weird "redefinition of s" error. For the 2nd and 3rd case there's still the same illformed AST message.

[Michael Schulte]

Hm - I see two versions of splitWhitespace - one is a proc, the other an iterator (how that can work I'm not sure...) Same goes for splitIterator
zero_functional does nothing special here - you can see the generated code when using the -->> operator. This should also give you an idea of what is going wrong!

as to "redefinition of s" -> that is why I wrote
readFile(...) .... --> createIter(s) and not let s = ... --> createIter(s).

Did you download the head of zero_functional.nim or are you using the nimble version 0.0.7 - I haven't updated that yet as I need to also test it.

[Vindaar]

Yep, two different versions. :)

Ah, nice. I'll check the -->> operator.

Yes, I'm aware. The redefinition error happens without the let s for this:

readFile(fileName).splitLines --> filter('#' notin it and it.len > 0) --> map(it.splitWhitespace) --> flatten() --> createIter(s)

I just cloned the repo locally and used the last commit of master.

[Michael Schulte]

So with -->> with your code I get the output:

# chk.nim:7
# readFile(fileName).splitLines.zfun:
#   filter('#' notin it and it.len > 0)
#   map(it.splitWhitespace)
#   flatten()
#   createIter(s)
iterator s(): auto =
  var idx = 0
  var idxFlatten198034 = -1
  for it0 in readFile(fileName).splitLines:
    zfParamChk("filter", '#' notin it0 and it0.len > 0, "cond", bool)
    if '#' notin it0 and it0.len > 0:
      let it1 = it0.splitWhitespace
      for flattened198036 in it1:
        let it2 = flattened198036
        idxFlatten198034 += 1
        let idx = idxFlatten198034
        discard (idx)
        yield it2
    idx += 1

So an iterator named s is created - which can be used with s() in later expressions.
Can't see where a redefinition is happening 😕

[Vindaar]

The output via expandMacros on the other hand is:

iterator s(): auto =
  var idx = 0
  var idxFlatten200045 = -1
  block :tmp200275:
      var
        __it__0
        :tmp
      :tmp = readFile("/home/schmidt/CastData/ExternCode/mpfit-nim/examples/data/half_life_muon.txt")
      var first = 0
      var last = 0
      block :tmp200283:
          while true:
            block :tmp200284:
                while last < len(:tmp) and not contains({'\c', '\n'}, :tmp[last]):
                  inc(last, 1)
            __it__0 = substr(:tmp, first, last - 1)
            discard
            if not contains(__it__0, '#') and
              ## "is greater" operator. This is the same as ``y < x``.
              0 < len(__it__0):
              let __it__1 = splitWhitespace(__it__0, -1)
              block :tmp200276:
                  var flattened200254
                  ## iterates over each item of `a`.
                  var i = 0
                  let L = len(__it__1)
                  block :tmp200277:
                      while i < L:
                        flattened200254 = __it__1[i]
                        let __it__2 = flattened200254
                        idxFlatten200045 += 1
                        let idx = idxFlatten200045
                        discard idx
                        yield __it__2
                        inc(i, 1)
                        if not (len(__it__1) == L):
                          failedAssertImpl("len(a) == L seq modified while iterating over it")
            idx += 1
            if
              ## "is greater or equals" operator. This is the same as ``y <= x``.
              len(:tmp) <= last:
              break :tmp200283
            if :tmp[last] == '\n':
              inc(last, 1)
            elif :tmp[last] == '\c':
              inc(last, 1)
              if last < len(:tmp) and :tmp[last] == '\n': inc(last, 1)
            first = last

test_zerof.nim(7, 1) template/generic instantiation from here
test_zerof.nim(8, 115) Error: redefinition of 's'

Line 7 is the expandMacros call and line 8 the line from my previous comment.

Can't find a redefinition there either.

[Vindaar]

From your snippet using -->> though, I'm still not sure whether the splitLines in use is actually an iterator or the proc

  for it0 in readFile(fileName).splitLines:

If the result of that is a seq this works the same as if the splitLines iterator is used. Interesting. :)

[Michael Schulte]

ahhhh.... so expandMacros - maybe when also using iterators - is the problem.

If you leave expandMacros out it should work!

This is like - maybe the 3rd or 4th bug concerning iterators in nim language I stumbled on... 😒

... as to splitLines etc.: the problem here using the iterator is when the result shall be determined automatically. This does not work for iterators - with the proc however it normally works fine.
... determining the result automatically also is only a problem when the result of the zero_functional calls is a sequence (or an iterator) itself again. When searching for a certain element for instance, the result of that operation is clear.

[Vindaar]

Like I said in the opening of this issue: yep, it works just fine without expandMacros. :) I just wanted to report this specifically as a thing happening if one wants to use expandMacros instead of -->> (which I wasn't really aware of before).

Ah, thanks. That makes sense!

[Michael Schulte]

OK - What I tried initially is to get your code compiled with the current version of zero_functional. Problem here was that the resulting sequence type could not be determined automatically. I was not aware that expandMacros was the actual problem.
So yes - please use -->> when debugging zero_functional (as also stated in the documentation) - as I said iterators in nim still got a lot of issues...

Also this sample code will give the same error result about the double definition:

import macros

expandMacros:
  iterator s(): auto =
    for i in 0..5:
      yield(i)

[Michael Schulte]

... and for your illformed AST problem see
nim-lang/Nim#7723
(stating 'illegal' but it really is the same illformed AST message)

[Vindaar]

Ahh, thanks a lot for your help! And sorry for assuming it's related to zero_functional in the first place. :) I'll close this.