plugnplay can't find implementors using different imports

Question

plugnplay can't find implementors using different imports

hugosenari opened this issue 12 years ago · comments

Hugo Sena Ribeiro commented 12 years ago

Hi,

I'm trying something like this:

I have:
main.py
modules/module.py
modules/notifiable.py

In notifiable.py
class _CoreModule(Interface):

In plugin.py:

from module import _CoreModule

In main.py:

from modules.module import _CoreModule

In this case plugnplay, can't find module:

Manager.implementors vars:

self.iface_implementors = dict: {<class 'module._CoreModule'>: [<configurable._Notifiable object at 0x2416310>}
interface =       InterfaceMeta: <class 'modules.module._CoreModule'>

The key has same content (and came from same file) but, from different imports.

In my tests this is normal, because dict use hash to match keys, and namespace matters to hash function

import dbus
from dbus import Bus

hash(dbus.bus)
#3051581

hash(Bus)
#3070785

id(Bus)
#49132560

id(dbus.bus)
#48825296

cmp(Bus, dbus.Bus)
#0    ( maybe solve this problem )

[]'s

Dalton Barreto · Answer 1 · Wed May 16 2012 01:28:19 GMT+0800 (China Standard Time)

This interesting. I see the point but I didn't fully undestand the example. You mentioned a plugin.py file, shouldn't it be notifiable.py?

I will try to write some code about this on the weekend. Thanks very much for the issue, I will take a look at this.

Dalton Barreto · Answer 2 · Wed May 16 2012 06:17:50 GMT+0800 (China Standard Time)

Hello @hugosenari,

Here is a first attempt to reproduce the problem. Could you please confirm that my code correctly checks for the bug?

Thanks.

Hugo Sena Ribeiro · Answer 3 · Wed May 16 2012 16:53:44 GMT+0800 (China Standard Time)

You can post here values for "interface" and self.iface_implementors from implementors for this two asserts?

= )

Dalton Barreto · Answer 4 · Wed May 16 2012 21:02:37 GMT+0800 (China Standard Time)

Sure, but independent of this the test passes. Wouldn't this be enough to ensure us that the code works?

Dalton Barreto · Answer 5 · Wed May 16 2012 21:09:18 GMT+0800 (China Standard Time)

Here is the output from the test run:

(plugnplay)daltonmatos@ShelbyCobra ~/src/plugnplay [15,issue/6]$ nosetests -s tests/loading.py 
..[<loading.MyImplementor object at 0x17f9b90>]
[<loading.MyImplementor object at 0x17f9b90>]
...
----------------------------------------------------------------------
Ran 5 tests in 0.002s

OK

If you want, try to write a test case that proves the error you found. I think this wat it will be easier to understand the problem.

Hugo Sena Ribeiro · Answer 6 · Thu May 17 2012 10:09:32 GMT+0800 (China Standard Time)

This week I can't run python apps. Only in next week. :(

The only way (for this week) that I can confirm are getting same (or equiv) output.

Your output show loading.MyImplementor in both assets, when in my are some like:
interfaces.myinterface.MyInterface
and myinterface.MyInterface
:)

Maybe structure matters:
init.py (load plugin and call methods, use this import: from myinterfaces.interface import MyInterface)
interfaces/myinterface.py (define interface)
interfaces/implementation.py (implement interface, use this import: from myinterface import MyInterface)

Dalton Barreto · Answer 7 · Sat May 19 2012 02:02:37 GMT+0800 (China Standard Time)

I think a code example would help more here. Could you please open a branch in your fork with the code that confirms the problem. Thanks.

Hugo Sena Ribeiro · Answer 8 · Fri May 25 2012 13:06:13 GMT+0800 (China Standard Time)

"Worked"

But to me (at window :( ):

Program send key error when remove _ init _

Solved adding one checker:
if mod_name in sys.modules:
del sys.modules[mod_name]

Hugo Sena Ribeiro · Answer 9 · Fri May 25 2012 14:47:20 GMT+0800 (China Standard Time)

Branch done: https://github.com/hugosenari/pig/tree/pigpnp

Here is my output from this version:

Interface _CoreModule loaded
C:\opt\devel\git\pigpnp\pig\core\interfaces
Interface _CoreModule loaded
Plugin for _CoreModule loaded: _CoreConfigurable
Plugin for _CoreModule loaded: _CoreNotifiable
Plugin for _CoreModule loaded: _CoreLoggable
Interface _CoreModule loaded
Plugin for _CoreModule loaded: _CoreNotifiable
Plugin for _CoreModule loaded: _CoreRunnable
Plugin for _CoreModule loaded: _CoreStarter
Plugin for _CoreModule loaded: _CoreVisible
size of _CoreModule implementors: []

Maybe the problem is for 'double import'

I do import _CoreModule to call 'main' functions

And import _CoreModule again by loadPlugins (_CoreModule are in same dir of my '_CorePlugins')

This version don't require nothing more then:

import plugnplay
import logging
from os.path import walk

(I run this at windows)

Try running pig.sh
If don't work, try running

cd pig\core\
python __.py

Thanks ;)

Dalton Barreto · Answer 10 · Sun May 27 2012 06:47:02 GMT+0800 (China Standard Time)

I think I understood what the problem is. Debugging your code, when I reach line 33 (_CoreModule.execute_modules(*args, **vargs)) I have this:

ipdb> _CoreModule
<class 'interfaces.module._CoreModule'>
ipdb> plugnplay.man.iface_implementors
{<class 'module._CoreModule'>: [<runnable._CoreRunnable object at 0x85b724c>,
<starter._CoreStarter object at 0x85b738c>, 
<visible._CoreVisible object at 0x85bc24c>, 
<indictable._CoreNotifiable object at 0x85b768c>, 
<notifiable._CoreNotifiable object at 0x85bc64c>, 
<loggable._CoreLoggable object at 0x85bca0c>,     
<configurable._CoreConfigurable object at 0x85bc98c>]}
ipdb>

I will try to discover why this same class gets imported differently. Maybe using cmp will indeed solve the problem, but I would lose the advantage of direct access, since I would probably have to iterate on the iface_implementors keys.

For the KeyError problem, I just opened another issue. #9.

Dalton Barreto · Answer 11 · Sun May 27 2012 06:52:21 GMT+0800 (China Standard Time)

One more thing I just discovered. No matter how I import a module it apperas to return the same canonical package, look:

ipdb> import interfaces
ipdb> interfaces.module._CoreModule
<class 'interfaces.module._CoreModule'>
ipdb> from interfaces import module
ipdb> module._CoreModule
<class 'interfaces.module._CoreModule'>
ipdb> _CoreModule
<class 'interfaces.module._CoreModule'>

I will try to check if this has something to do with the fact that python does not import one module twice. I will keep you posted.

Dalton Barreto · Answer 12 · Sun May 27 2012 06:57:30 GMT+0800 (China Standard Time)

I opened the python shell many times and on each of them I imported _CoreModule differently. All times it returned the same canonical package name: interfcaes.module._CoreModule.

Now I have to discover why plugnplay is concluding the wrong package name when storing the implementors for an plugnplay.Interface.

Dalton Barreto · Answer 13 · Sun May 27 2012 07:13:25 GMT+0800 (China Standard Time)

I made a simple change to your code. I imported the plugins using normal python code. I removed the __init__() method from the Pig class and added one more import:

import interfaces.configurable

and the output is correct:

(plugnplay)daltonmatos@jetta ~/projetos/contrib/pig/pig/core [78,pigpnp]$ python __init__.py 
Interface _CoreModule loaded
Plugin for _CoreModule loaded: _CoreConfigurable
size of _CoreModule (<class 'interfaces.module._CoreModule'>) implementors:     [<interfaces.configurable._CoreConfigurable object at 0xf70efe0c>]
plugnplay.man.iface_implementors:  {<class 'interfaces.module._CoreModule'>:     [<interfaces.configurable._CoreConfigurable object at 0xf70efe0c>]}
(plugnplay)daltonmatos@jetta ~/projetos/contrib/pig/pig/core [79,pigpnp]$

Something is very wrong when plugnplay imports the modules. Maybe there is something to do with the fact that all Interfaces are also inside the same python package of the plugins, and plugnplay imports them. Maybe when plugnplay runs __import__('module') is where _CoreModule changes its canonical package name from interfaces.module._CoreModule to only module._CoreModule.

I will continue investigating! =)

Dalton Barreto · Answer 14 · Sun May 27 2012 07:43:07 GMT+0800 (China Standard Time)

This is interesting: The pwd is pig/core here.

In [1]: import sys

In [2]: sys.path.insert(0, 'interfaces')

In [3]: from interfaces import module
Interface _CoreModule loaded

In [5]: module._CoreModule
Out[5]: interfaces.module._CoreModule

In [6]: m = __import__('module')
Interface _CoreModule loaded

In [7]: m
Out[7]: <module 'module' from 'interfaces/module.pyc'>

In [8]: m._CoreModule
Out[8]: module._CoreModule

In [9]: module._CoreModule
Out[9]: interfaces.module._CoreModule

In [10]:

Dalton Barreto · Answer 15 · Sun May 27 2012 09:21:16 GMT+0800 (China Standard Time)

I think the problem is because the folder of the python package is added as a plugin folder in plugnplay. In this case we have the folder interfaces (that, at the same time, is a python package) with this structure:

interfaces
  - __init__.py
  - module.py
  - configurable.py

In this case, the implemented interfaces are inside the module.py module. The problem is that, since we add interfaces/ as a plungplay plugin dir, plugnplay adds this folder to the sys.path. This is how it can import the modules inside it. But here is the problem: As soon as plugnplay adds this folder to sys.path, the original interfaces package does not exists any more, so any class inside this package will never have its package name including the original interfaces part.

We are comparing the plugnplay imports with an import from the python perspective (where the original interfaces package still exists!) and that's why we see difference between the imports. The import made with regular python code returns interfaces.module._CoreModule but the import made by plugnplay (where interfaces package is vanished due to its addition do sys.path) is just module._CoreModule.

Dalton Barreto · Answer 16 · Sun May 27 2012 09:38:50 GMT+0800 (China Standard Time)

Honestly, I don't know how to solve this. Maybe plugnplay cloud detect that the plugindir is also a python package, and in this case is adds do sys.path just the top level folder, not the plugin folder itself.

In our example (your example branch), it would add pig/core do sys.path and not pig/core/interfaces.

This could work. =)

Dalton Barreto · Answer 17 · Sun May 27 2012 10:02:40 GMT+0800 (China Standard Time)

The problem occurs if the implemented interface is imported before the implementors. This is a complete output from my debug session: First I loaded the plugins without importing the interface and then I first imported the interface.

(plugnplay)daltonmatos@jetta ~/projetos/plugnplay/tests [149,issue/6]$ py
Python 2.7.2 (default, Oct 28 2011, 21:14:54) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import sys

In [2]: import plugnplay

In [3]: plugnplay.set_plugin_dirs('allinone')

In [4]: plug
plugin-metaclass.py   plugin-metaclass.pyc  plugin/               plugnplay             

In [4]: plugnplay.load_plugins()

In [5]: plugnplay.man.iface_implementors
Out[5]: {interface.AllInOne: [<module.AllInOnePlugin at 0x85b7f6c>]}

In [6]: from allinone.interface import AllInOne

In [7]: AllInOne
Out[7]: interface.AllInOne

In [8]: AllInOne.implementors()
Out[8]: [<module.AllInOnePlugin at 0x85b7f6c>]

In [9]: 
Do you really want to exit ([y]/n)?

Now, I repeat the same process, but first I import the interface, from allinone.interface.

(plugnplay)daltonmatos@jetta ~/projetos/plugnplay/tests [150,issue/6]$ py
Python 2.7.2 (default, Oct 28 2011, 21:14:54) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from allinone.interface import AllInOne

In [2]: AllInOne
Out[2]: allinone.interface.AllInOne

In [3]: import plugnplay

In [4]: plugnplay.set_plugin_dirs('allinone')

In [5]: plugnplay.load_plugins()

In [6]: plugnplay.man.iface_implementors
Out[6]: {interface.AllInOne: [<module.AllInOnePlugin at 0x85b010c>]}

In [7]: AllInOne
Out[7]: allinone.interface.AllInOne

In [8]: AllInOne.implementors()
Out[8]: []

In [9]:

I still think that detecting the presence of a python package and adding just the top level directory could fix this behavior.

Hugo Sena Ribeiro · Answer 18 · Mon May 28 2012 03:06:49 GMT+0800 (China Standard Time)

How about make your own key?
I dislike my solution, don't know porformance of 'getfile', maybe one other property can be used for hash, or do it on class load to don't calculate again every time I call implementators.

Dalton Barreto · Answer 19 · Mon May 28 2012 03:15:40 GMT+0800 (China Standard Time)

That's exactly what I'm doing right now! I will push the solution in a moment. I'm using the canonical package name as the hash. Since I solved the problem of the import, both imported modules now returns interfaces.module._CoreModule, the comparison works now.

I'm just doing final refactorings and will push the code.

Dalton Barreto · Answer 20 · Mon May 28 2012 03:33:06 GMT+0800 (China Standard Time)

Running your code again shows the correct output:

(plugnplay)daltonmatos@jetta ~/projetos/contrib/pig [32,pigpnp]$ python pig/core/__init__.py 
Interface _CoreModule loaded
/home/daltonmatos/projetos/contrib/pig/pig/core/interfaces
Interface _CoreModule loaded
Plugin for _CoreModule loaded: _CoreRunnable
Plugin for _CoreModule loaded: _CoreStarter
Plugin for _CoreModule loaded: _CoreVisible
Plugin for _CoreModule loaded: _CoreNotifiable
Plugin for _CoreModule loaded: _CoreNotifiable
Plugin for _CoreModule loaded: _CoreLoggable
Plugin for _CoreModule loaded: _CoreConfigurable
_CoreVisible module loaded
size of _CoreModule implementors: [
<interfaces.runnable._CoreRunnable object at 0x80f764c>,
<interfaces.starter._CoreStarter object at 0x80f782c>, 
<interfaces.visible._CoreVisible object at 0x80f79ec>, 
<interfaces.indictable._CoreNotifiable object at 0x80f7bec>, 
<interfaces.notifiable._CoreNotifiable object at 0x80f7d6c>, 
<interfaces.loggable._CoreLoggable object at 0x80f7f2c>, 
<interfaces.configurable._CoreConfigurable object at 0x80fb14c>]
(plugnplay)daltonmatos@jetta ~/projetos/contrib/pig [33,pigpnp]$

Thank you VERY MUCH for your help. It was indeed a very tricky bug but also was very fun to solve it!!! Thank you! I will release a new version of plugnplay soon.