Altered scope in function definitions when going through py4cl

[Eric Schulte]

Hi, I'm confused by how the scope may be changed when I'm executing python code through py4cl, specifically when trying to reference global objects or functions from within a function definition. Here's a small example demonstrating the problem.

I have a very simple tracing library named simple_tracer.py:

trace_points = []

def record(n):
    trace_points.append(n)

def clear():
    global trace_points
    trace_points = []

def trace():
    return trace_points

I have a simple test script that uses that library named test.py:

import simple_tracer
simple_tracer.clear()

def add_eight_or_four(n):
  simple_tracer.record(13)
  if(n > 8):
    simple_tracer.record(8)
    return n + 4
  else:
    simple_tracer.record(12)
    return n + 7

print(add_eight_or_four(2))
print(simple_tracer.trace())

This executes as expected in the python repl or at the command line:

$ python test.py
9
[13, 12]

However, when I try to run this through py4cl, I get the following error that "name 'simple_tracer' is not defined".

(python-exec "import simple_tracer
simple_tracer.clear()

def add_eight_or_four(n):
  simple_tracer.record(13)
  if(n > 8):
    simple_tracer.record(8)
    return n + 4
  else:
    simple_tracer.record(12)
    return n + 7

print(add_eight_or_four(2))
print(simple_tracer.trace())
")

yields

Python error: "name 'simple_tracer' is not defined"
   [Condition of type PYTHON-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {10034A9EC3}>)

Backtrace:
  0: (PY4CL::DISPATCH-MESSAGES #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {1002A39D83}>)
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (PYTHON-EXEC "import simple_tracer ..)
  2: (EVAL (PYTHON-EXEC "import simple_tracer ..)
  3: (SWANK::EVAL-REGION "(python-exec \"import simple_tracer ..)

Any explanation or advice would be much appreciated.

[Ben Dudson]

Hi @eschulte thanks for the bug report and short example. As a quick check, I tried running the same string through the python exec function from an ipython shell. That seemed to work as expected.
Could it be something to do with the working directory, so that the python subprocess can't find the file to import? Could you please try:

(python-exec "import os; print(os.getcwd())")

[Eric Schulte]

Thanks for the quick reply! It isn't an issue with the load path or directory. When I run the above I get my current working directory in which the simple_tracer file lives.

I was able to demonstrate similar behavior just using a global array instead of a global module. Let me see if I can resurrect that example... Alright, so with this file simple_test.py:

trace = []

def add_eight_or_four(n):
  trace.append(13)
  if(n > 8):
    trace.append(8)
    return n + 4
  else:
    trace.append(12)
    return n + 7

print(add_eight_or_four(2))
print(trace)

I get this output at the command line:

$ python simple_test.py
9
[13, 12]

and I get this result in the REPL with python-exec:

(python-exec "trace = []

def add_eight_or_four(n):
  trace.append(13)
  if(n > 8):
    trace.append(8)
    return n + 4
  else:
    trace.append(12)
    return n + 7

print(add_eight_or_four(2))
print(trace)
")

gives this error:

Python error: "name 'trace' is not defined"
   [Condition of type PYTHON-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1003489EC3}>)

Backtrace:
  0: (PY4CL::DISPATCH-MESSAGES #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {100486A123}>)
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (PYTHON-EXEC "trace = [] ..)
  2: (EVAL (PYTHON-EXEC "trace = [] ..)
...

Could this have something to do with the use of Python's eval and exec functions limiting the scope?

[Eric Schulte]

Running this in the Python REPL is illuminating.

>>> exec("""
... trace = []
... 
... def add_eight_or_four(n):
...   trace.append(13)
...   if(n > 8):
...     trace.append(8)
...     return n + 4
...   else:
...     trace.append(12)
...     return n + 7
... 
... print(add_eight_or_four(2))
... print(trace)
... """)
9
[13, 12]
>>> exec("""trace = []
... 
... def add_eight_or_four(n):
...   trace.append(13)
...   if(n > 8):
...     trace.append(8)
...     return n + 4
...   else:
...     trace.append(12)
...     return n + 7
... 
... print(add_eight_or_four(2))
... print(trace)""", {}, {})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 12, in <module>
  File "<string>", line 4, in add_eight_or_four
NameError: name 'trace' is not defined

It looks like passing in the optional global and local scope arguments to exec removes some default scope. I wonder if there is a way to combine your explicit eval_globals and eval_locals with the Python defaults?

[Deleted user]

This issue appears to be in how locals and globals are handled in python3.x, as explained in this StackOverflow answer. In short, the behavior and role of locals appears to have changed between 2.x and 3.x; in 3.x, the documentation now states:

Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.

As of Python 3.x, it appears that values stored in the locals dictionary are not available for during subsequent calls to exec.
The link above suggests the simplest solution is to use a single dictionary:

>>> exec("""
... def foo():
...   return 5
... 
... def bar():
...   return foo() + foo()
... 
... print(bar())""", {})
10

This reuses the same dictionary for both locals and globals, providing saner semantics and avoiding the problem.