Add support for IDA 7.4 & Python 3

Question

Add support for IDA 7.4 & Python 3

tmr232 opened this issue 5 years ago · comments

Loading the plugin in IDA 7.4 Python 3 requires some minor tweaking:

Run 2to3
Add a None check to https://github.com/eset/ipyida/blob/master/ipyida/kernel.py#L29 as sys.__stdout__ is None for some reason.

Once done, the plugin loads and runs. However, it completely fails whenever any exception is thrown:

Jupyter QtConsole 4.5.1
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.

few
ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-6e4eba6e7240>", line 1, in <module>
    few
NameError: name 'few' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 2039, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'NameError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 1101, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 319, in wrapped
    return f(*args, **kwargs)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 353, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__plugin_loader' from ''> is a built-in module
---------------------------------------------------------------------------

Note that __plugins__plugin_loader is the name of my plugin loader (plugin_loader). If this is used without the plugin loader - the result is the same, but a different module name appears.

I believe this should probably be fixed in IPython, but thought it important to report here as well.

Marc-Etienne M.Léveillé · Answer 1 · Mon Oct 07 2019 09:51:34 GMT+0800 (China Standard Time)

Ho yes supporting 7.4 and Python 3 is a must. I'll ask for the beta now so I can test and update this week. I'm pretty such the switch to Py3 will break a few things...

There are also a few interesting things I left unreleased in the master branch (including somewhat-support for dark mode). I'll tackle this and do a release soon.

Arnaud Diederen · Answer 2 · Mon Oct 07 2019 17:40:37 GMT+0800 (China Standard Time)

Hi @marc-etienne, @tmr232 ,

(This is arnaud-at-hex-rays.com)

Although it is apparently not quite clear what the problem is at this point, please let me know if there's something to be done on IDA's side to support IPython, once you know what it is.
I cannot spend much time on IPython itself, but if there's something IDAPython does poorly which causes these failures, we're definitely willing to rectify that.

Marc-Etienne M.Léveillé · Answer 3 · Mon Oct 07 2019 22:33:23 GMT+0800 (China Standard Time)

Thanks @aundro !

I've requested the beta a few hours ago. Once I have a copy I'll test it out and update the status here.

Marc-Etienne M.Léveillé · Answer 4 · Mon Oct 21 2019 23:02:34 GMT+0800 (China Standard Time)

I started looking into this last Friday using the stable 7.4 macOS release. I didn't need to change anything in IPyIDA source to make it works (yay!).

However, I am able to reproduce the ERROR:root:Internal Python error in the inspect module [...] TypeError: <module '****' from ''> is a built-in module.

I'm still trying to find the root cause. Here are some observation:

It only happens on a NameError, try raising a syntax SyntaxError works as expected
Something is trying to call getfile() using a built-in module, but that seems to be in IPython
After 4 or 5 NameErrors, it goes back to normal and exceptions are properly displayed.
Each TypeError from getfile() shows a different module name (__main__, __plugins__ipyida, ...)

I will continue investigating.

Marc-Etienne M.Léveillé · Answer 5 · Mon Oct 21 2019 23:05:41 GMT+0800 (China Standard Time)

About this:

Add a None check to https://github.com/eset/ipyida/blob/master/ipyida/kernel.py#L29 as sys.stdout is None for some reason.

@tmr232 : Under what OS?

Marc-Etienne M.Léveillé · Answer 6 · Mon Oct 21 2019 23:56:13 GMT+0800 (China Standard Time)

I was able to reproduce without IPyIDA. Simply by doing the following:

Open IDA 7.4 (no need for opening a file)
In the console, type: import inspect; inspect.stack()

Output:

Python>import inspect
Python>inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__main__' (built-in)> is a built-in module
Python>inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__ipyida' from ''> is a built-in module
Python>inspect.stack()
Python>print(inspect.stack())
[FrameInfo(frame=<frame at 0x1186b5868, file '<string>', line 1, code <module>>, filename='<string>', lineno=1, function='<module>', code_context=None, index=None)]

I'm not sure if this is a could be a bug in Python's inspect.py or something funky IDAPython does.
@aundro : let me know if you think of anything.

Marc-Etienne M.Léveillé · Answer 7 · Tue Oct 22 2019 03:07:52 GMT+0800 (China Standard Time)

I found the root cause and was able to fix the issue with a single line change in IDAPython. @aundro expect a pull request very soon.

Arnaud Diederen · Answer 8 · Tue Oct 22 2019 15:41:55 GMT+0800 (China Standard Time)

Good to hear. Hopefully that single line change won't have unintended consequences on other scripts.

Tamir Bahar · Answer 9 · Tue Oct 22 2019 19:39:52 GMT+0800 (China Standard Time)

This sounds like the same issue I was seeing with idaapi.require. Happy to hear a solution exists :)

Arnaud Diederen · Answer 10 · Tue Oct 22 2019 19:47:22 GMT+0800 (China Standard Time)

Running import inspect; inspect.stack() in a fresh new IDA 7.4 session works for me (both on windows & linux.)

I wonder what's causing the change in behavior on your box…

Tamir Bahar · Answer 11 · Tue Oct 22 2019 20:09:25 GMT+0800 (China Standard Time)

Works on a fresh install, but seems to fail if I have any Python plugins loaded.

Arnaud Diederen · Answer 12 · Tue Oct 22 2019 21:04:30 GMT+0800 (China Standard Time)

I just:

added the SDK's idasdk74/plugins/script_plg/pyplugin.py to my IDA 7.4 install,
started IDA, and ran the plugin

-> import inspect; inspect.stack() still works for me.

Tamir Bahar · Answer 13 · Tue Oct 22 2019 21:27:41 GMT+0800 (China Standard Time)

I think the plugin flags have to be ida_idaapi.PLUGIN_FIX for reproduction.

This is what I get:

Python>import inspect; inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__pyplugin' from ''> is a built-in module

Arnaud Diederen · Answer 14 · Tue Oct 22 2019 22:25:51 GMT+0800 (China Standard Time)

@tmr232 ida_idaapi.PLUGIN_FIX with the example pyplugin.py, doesn't change anything for me.

I'd really like to be able to reproduce this, before I accept the PR. Can one of you guys put me on the right track?
If I understand @marc-etienne 's fix correctly, IDAPython_ExecScript could be called with globals coming from a builtin module. How can I reproduce this?

Tamir Bahar · Answer 15 · Tue Oct 22 2019 22:40:52 GMT+0800 (China Standard Time)

@marc-etienne Using your patch I no longer get the exception, but IPython dies anyhow. I get a regular exception the first time, then every line results in:
ERROR: execution aborted

Marc-Etienne M.Léveillé · Answer 16 · Tue Oct 22 2019 22:48:34 GMT+0800 (China Standard Time)

@tmr232: Try installing IPyIDA from the master branch

Marc-Etienne M.Léveillé · Answer 17 · Tue Oct 22 2019 23:10:08 GMT+0800 (China Standard Time)

Sorry for the cliffhanger yesterday :D. Basically, having __file__ set to an empty string is a edge case not handled properly in inspect.py. After IDAPython_ExecScript is executed, sys.modules["__main__"].__file__ (or a __plugin_XXX namespace) may end up being empty string. This will stay persistant. In inspect.py:getmodule(), the code will reach line 739 which iterates over all modules and enventually call getfile() using the built-in module, which raises a TypeError.

Yeah this is hard to follow :P, but seems like not tempering with __file__ too much avoid this error.

@aundro : can you tell me which Python version you have installed?

Arnaud Diederen · Answer 18 · Tue Oct 22 2019 23:17:33 GMT+0800 (China Standard Time)

Windows: 3.7.4
Linux: 3.5.3

Tamir Bahar · Answer 19 · Wed Oct 23 2019 20:04:44 GMT+0800 (China Standard Time)

@marc-etienne Tested with latest, and it works.
Seems like my issue was having a newer version of IPython.

Now that we're using Python3, do you have any idea what it'd take to migrate to latest IPython/Jupyter?

Marc-Etienne M.Léveillé · Answer 20 · Thu Oct 24 2019 03:31:00 GMT+0800 (China Standard Time)

I added some code that works around the issue without having to modify IDAPython, so it should work out of the box on current IDA 7.4.

I will test on other platforms (more specifically on Windows), test if the install script needs changes too and do a release next week.

Seems like my issue was having a newer version of IPython.
Now that we're using Python3, do you have any idea what it'd take to migrate to latest IPython/Jupyter?

I also tested upgrading ipykernel to version >= 5 and I have the same ERROR: execution aborted error after a few successful commands. I tried finding the cause of that error for a few hours and I hasn't successful so far 😞.

Since this is not required for IDA 7.4 and Python 3 support, I will track this in #26.

Marc-Etienne M.Léveillé · Answer 21 · Tue Nov 05 2019 00:24:35 GMT+0800 (China Standard Time)

v1.4 is out, 🍾

Fritz Reese · Answer 22 · Fri Nov 13 2020 04:56:06 GMT+0800 (China Standard Time)

FYI: The fix for this breaks IDA 6.xx, where IDAPython_ExecScript takes only two arguments.