Failed to detect AST nodes in Windows ipython for verb calls

Question

Failed to detect AST nodes in Windows ipython for verb calls

GitHunter0 opened this issue 2 years ago · comments

GitHunter0 commented 2 years ago

Feature Type

Adding new functionality to datar
Changing existing functionality in datar
Removing existing functionality in datar

Problem Description

The code below

    import datar.all as d
    import pandas as pd
    new_name = "y"
    pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})

generates the following warning message:

c:\Users\me\miniconda3\envs\cenv\lib\site-packages\pipda\utils.py:89: PipeableCallCheckWarning:
Failed to detect AST node calling `rename`, assuming a piping call.

Feature Description

Being able to suppress the warning message above.

Additional Context

No response

GitHunter0 commented 2 years ago

For sure:

pwwang · Answer 1 · Wed Jan 18 2023 03:29:48 GMT+0800 (China Standard Time)

Could you also indicate the environment where you were running the code? i.e. python script, jupyter, python REPL, etc

GitHunter0 · Answer 2 · Wed Jan 18 2023 06:23:51 GMT+0800 (China Standard Time)

For sure @pwwang , I'm using in a .py file in VScode (via its Interactive Window, which uses jupyter kernel).

pwwang · Answer 3 · Wed Jan 18 2023 06:40:31 GMT+0800 (China Standard Time)

I can run it with no problem:

here is the version info:

Version: 1.74.3 (system setup)
Commit: 97dec172d3256f8ca4bfb2143f3f76b503ca0534
Date: 2023-01-09T16:59:02.252Z
Electron: 19.1.8
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.25276
Sandboxed: No

vscode-jupyter: v2022.11.1003412109

GitHunter0 · Answer 4 · Wed Jan 18 2023 07:07:48 GMT+0800 (China Standard Time)

The problem was indeed with VScode, I updated it to the last version and that warning message disappeared, thank you @pwwang .

GitHunter0 · Answer 5 · Thu Jan 19 2023 00:22:22 GMT+0800 (China Standard Time)

@pwwang , I checked that the issue was solved only for python 3.8 (after I updated datar and restarted VScode).
All conda environments I tested with python 3.9 or 3.10 that message appears, even directly in the command prompt (without VScode and ipython), it's very weird...

GitHunter0 · Answer 6 · Thu Jan 19 2023 00:35:31 GMT+0800 (China Standard Time)

Steps to reproduce the issue.
Go to Windows OS powershell and type:

conda create --name py3.9 python=3.9 
conda activate py3.9 
pip install datar[pandas] 
python

Then run:

    import datar.all as d
    import pandas as pd
    new_name = "y"
    pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})

It will display this message:

C:\Users\me\miniconda3\envs\py3.9\lib\site-packages\pipda\utils.py:89: PipeableCallCheckWarning: Failed to detect AST node calling `rename`, assuming a piping call.
  warnings.warn(

pwwang · Answer 7 · Thu Jan 19 2023 01:31:47 GMT+0800 (China Standard Time)

This is expected.
There is some description here: https://pwwang.github.io/pipda/piping/#fallbacks-when-ast-node-detection-fails

Tracing executing node is not available in the raw python REPL.
How about ipython or bpython?

GitHunter0 · Answer 8 · Thu Jan 19 2023 01:52:53 GMT+0800 (China Standard Time)

I could not install bpython to test, but with ipython the same issue occurs and since VScode relies on ipython for its Interactive Window, it inherits the same behavior.

GitHunter0 · Answer 9 · Thu Jan 19 2023 01:57:20 GMT+0800 (China Standard Time)

Maybe an option to temporarily silent that message would suffice as a workaround (in the context of a function, we could turn off in the beginning and turn on in the end).

pwwang · Answer 10 · Thu Jan 19 2023 02:07:16 GMT+0800 (China Standard Time)

You can silence it by:

pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'}, __ast_fallback="piping")

It's kinda wired that ipython shows the same issue.

Could you provide datar.get_versions() and also the version of your python?

GitHunter0 · Answer 11 · Thu Jan 19 2023 02:43:21 GMT+0800 (China Standard Time)

Awesome, __ast_fallback="piping" works.

Could you provide datar.get_versions() and also the version of your python?

For sure:

python      : 3.9.16 (main, Jan 11 2023, 16:16:36) [MSC v.1916 64 bit (AMD64)]
datar       : 0.11.1
simplug     : 0.2.2
executing   : 1.2.0
pipda       : 0.11.0
datar-numpy : 0.1.0
numpy       : 1.24.1
datar-pandas: 0.2.0
pandas      : 1.5.2

The python version installed on my Windows OS is 3.9.4.

pwwang · Answer 12 · Thu Jan 19 2023 02:45:36 GMT+0800 (China Standard Time)

The python version installed on my Windows OS is 3.9.4.

Sorry, I meant ipython.

pwwang · Answer 13 · Thu Jan 19 2023 02:52:52 GMT+0800 (China Standard Time)

pipda v0.11.1 was just released. With it, you can change the default fallback of a verb:

# importing ...
d.rename.ast_fallback  = "piping"
# So you don't need to pass "__ast_fallback" to every call
new_name1 = 'y'
new_name2 = 'z'
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name1: 'x'})
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name2: 'x'})

GitHunter0 · Answer 14 · Thu Jan 19 2023 02:59:53 GMT+0800 (China Standard Time)

pipda v0.11.1 was just released. With it, you can change the default fallback of a verb:

Very nice to know.

GitHunter0 · Answer 15 · Thu Jan 19 2023 03:00:31 GMT+0800 (China Standard Time)

Sorry, I meant ipython.

The ipython version is 8.8.0

pwwang · Answer 16 · Thu Jan 19 2023 03:15:09 GMT+0800 (China Standard Time)

I don't see a critical difference between your environment and mine below:

Could you also take a similar snapshot?

pwwang · Answer 17 · Thu Jan 19 2023 03:53:17 GMT+0800 (China Standard Time)

Looks like the only difference is the platform. I can reproduce the issue on Windows. The snapshot I took was on Linux (WSL).
I'll dig deeper to see if there is a fix.

pwwang · Answer 18 · Thu Jan 19 2023 04:22:17 GMT+0800 (China Standard Time)

Looks like it is due to a python bug on Windows. AST nodes cannot be detected when we are trying to deconstruct dicts as keyword-arguments.

new_name = 'y'
# Warns
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})
# But this is Okay
pd.DataFrame(dict(x=[2])) >> d.rename(y='x')

GitHunter0 · Answer 19 · Thu Jan 19 2023 05:04:28 GMT+0800 (China Standard Time)

Great, at least now we know the root cause.

pwwang · Answer 20 · Tue Mar 07 2023 04:19:20 GMT+0800 (China Standard Time)

Close it for now since this problem can not be solved in this repo.

A temporarily solution is to pass __ast_fallback to bypass the AST node detection:

pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'}, __ast_fallback="piping")

See more details here: https://pwwang.github.io/pipda/piping/#fallbacks-when-ast-node-detection-fails