Support for napalm v4.x?

Question

Support for napalm v4.x?

ZPrimed opened this issue 2 years ago · comments

napalm-ros currently requires napalm == 3.*
Development in the napalm world seems to move pretty quick and they are up to v4.0.0 as current stable (this does require python 3.7+, but 3.0.0 was 3.6+ so I don't think it's a drastic python change like 2->3)

Any chance that napalm-ros can be bumped to napalm 4.x?

I'm not much of a programmer so I don't really know how to do this - my installation of both napalm and napalm-ros are coming from pip and are being pulled in as a local_requirement for NetBox, so I'm not exactly sure how to force napalm-ros to install after I've required a higher version of napalm.

Brian Candler · Answer 1 · Wed Jul 20 2022 16:29:19 GMT+0800 (China Standard Time)

And the fundamental problem is:

installing napalm-ros gives you napalm 3.x (currently 3.3.1)
napalm 3.3.1 depends on netmiko>=3.1.0
...which allows it to pull in the latest netmiko 4.1.1
...but napalm 3 doesn't work with netmiko 4 (backwards-incompatible API change: they removed netmiko.ssh_exception)

Hence napalm crashes and burns as soon as you import it:

(venv) root@netbox3:~# python
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import napalm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/__init__.py", line 11, in <module>
    import napalm.iosxr
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/iosxr/__init__.py", line 22, in <module>
    from napalm.iosxr.iosxr import IOSXRDriver  # noqa
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/iosxr/iosxr.py", line 28, in <module>
    from napalm.pyIOSXR import IOSXR
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/__init__.py", line 26, in <module>
    from napalm.pyIOSXR.iosxr import IOSXR
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/iosxr.py", line 38, in <module>
    from netmiko.ssh_exception import NetMikoTimeoutException
ModuleNotFoundError: No module named 'netmiko.ssh_exception'
>>>

Łukasz Kostka · Answer 2 · Sat Jul 23 2022 05:13:56 GMT+0800 (China Standard Time)

With:

pip install 'napalm==3.*'
Successfully installed napalm-3.4.1 netmiko-3.4.0

That's not all. Napalm 4.x broke/fixed tests in this PR. I need to handle this in code and on CI.

Brian Candler · Answer 3 · Mon Jul 25 2022 00:32:25 GMT+0800 (China Standard Time)

pip install 'napalm==3.*'
Successfully installed napalm-3.4.1 netmiko-3.4.0

But if you install both napalm and napalm-ros in the same command:

virtualenv venv
. venv/bin/activate
pip install --upgrade pip
pip install napalm napalm-ros

I get:

Collecting napalm
  Downloading napalm-4.0.0-py2.py3-none-any.whl (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.2/266.2 kB 8.0 MB/s eta 0:00:00
Collecting napalm-ros
  Using cached napalm_ros-1.0.1-py3-none-any.whl
Collecting netmiko>=4.0.0
  Downloading netmiko-4.1.1-py3-none-any.whl (192 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 192.8/192.8 kB 10.6 MB/s eta 0:00:00
...

and then, with the venv still activated:

$ python
Python 3.10.4 (main, Apr 26 2022, 19:42:59) [Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import napalm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brian/tmp/zzz/venv/lib/python3.10/site-packages/napalm/__init__.py", line 11, in <module>
    import napalm.iosxr
  File "/Users/brian/tmp/zzz/venv/lib/python3.10/site-packages/napalm/iosxr/__init__.py", line 22, in <module>
    from napalm.iosxr.iosxr import IOSXRDriver  # noqa
  File "/Users/brian/tmp/zzz/venv/lib/python3.10/site-packages/napalm/iosxr/iosxr.py", line 28, in <module>
    from napalm.pyIOSXR import IOSXR
  File "/Users/brian/tmp/zzz/venv/lib/python3.10/site-packages/napalm/pyIOSXR/__init__.py", line 26, in <module>
    from napalm.pyIOSXR.iosxr import IOSXR
  File "/Users/brian/tmp/zzz/venv/lib/python3.10/site-packages/napalm/pyIOSXR/iosxr.py", line 38, in <module>
    from netmiko.ssh_exception import NetMikoTimeoutException
ModuleNotFoundError: No module named 'netmiko.ssh_exception'
>>>

$ pip freeze | egrep 'napalm|miko'
napalm==3.3.1
napalm-ros==1.0.1
netmiko==4.1.1
paramiko==2.11.0

There is a long-standing issue with pip when multiple packages depend on the same underlying package but with different version constraints :-(

Łukasz Kostka · Answer 4 · Sat Jul 30 2022 01:58:46 GMT+0800 (China Standard Time)

First you've written that pip installs:

Collecting napalm
  Downloading napalm-4.0.0-py2.py3-none-any.whl (266 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 266.2/266.2 kB 8.0 MB/s eta 0:00:00
Collecting napalm-ros
  Using cached napalm_ros-1.0.1-py3-none-any.whl
Collecting netmiko>=4.0.0
  Downloading netmiko-4.1.1-py3-none-any.whl (192 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 192.8/192.8 kB 10.6 MB/s eta 0:00:00
...

Later:

$ pip freeze | egrep 'napalm|miko'
napalm==3.3.1
napalm-ros==1.0.1
netmiko==4.1.1
paramiko==2.11.0

So which is true ?

Brian Candler · Answer 5 · Sat Jul 30 2022 02:34:47 GMT+0800 (China Standard Time)

Sorry, that was a copy-paste error.

When I did pip install napalm-ros, I got a consistent set of packages. When I did pip install napalm napalm-ros, I got the conflict (napalm 3.x with netmiko 4.x)

Starting with fresh virtualenvs: here is pip install napalm-ros:

MacBook-Pro-5:tmp $ mkdir test1
MacBook-Pro-5:tmp $ cd test1
MacBook-Pro-5:test1 $ virtualenv venv
created virtual environment CPython3.10.4.final.0-64 in 461ms
  creator CPython3Posix(dest=/Users/brian/tmp/test1/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/brian/Library/Application Support/virtualenv)
    added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
MacBook-Pro-5:test1 $ . venv/bin/activate
(venv) MacBook-Pro-5:test1 $ pip install napalm-ros
Collecting napalm-ros
  Using cached napalm_ros-1.0.1-py3-none-any.whl
Collecting librouteros==3.*
  Using cached librouteros-3.2.1-py3-none-any.whl
Collecting napalm==3.*
  Using cached napalm-3.4.1-py2.py3-none-any.whl (256 kB)
Collecting future
  Using cached future-0.18.2-py3-none-any.whl
Collecting netaddr
  Using cached netaddr-0.8.0-py2.py3-none-any.whl (1.9 MB)
Collecting requests>=2.7.0
  Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting ncclient
  Using cached ncclient-0.6.13-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=38.4.0 in ./venv/lib/python3.10/site-packages (from napalm==3.*->napalm-ros) (62.6.0)
Collecting pyYAML
  Using cached PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl (197 kB)
Collecting paramiko>=2.6.0
  Using cached paramiko-2.11.0-py2.py3-none-any.whl (212 kB)
Collecting scp
  Using cached scp-0.14.4-py2.py3-none-any.whl (8.6 kB)
Collecting pyeapi>=0.8.2
  Using cached pyeapi-0.8.4-py3-none-any.whl
Collecting netmiko<4.0.0,>=3.3.0
  Using cached netmiko-3.4.0-py3-none-any.whl (178 kB)
Collecting jinja2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting lxml>=4.3.0
  Using cached lxml-4.9.1-cp310-cp310-macosx_12_0_x86_64.whl
Collecting textfsm
  Using cached textfsm-1.1.3-py2.py3-none-any.whl (44 kB)
Collecting junos-eznc>=2.6.3
  Downloading junos_eznc-2.6.5-py2.py3-none-any.whl (197 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 197.4/197.4 kB 5.9 MB/s eta 0:00:00
Collecting cffi>=1.11.3
  Using cached cffi-1.15.1-cp310-cp310-macosx_10_9_x86_64.whl (179 kB)
Collecting ciscoconfparse
  Using cached ciscoconfparse-1.6.40-py3-none-any.whl (136 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting transitions
  Using cached transitions-0.8.11-py2.py3-none-any.whl (83 kB)
Collecting pyparsing
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting yamlordereddictloader
  Using cached yamlordereddictloader-0.4.0-py3-none-any.whl
Collecting pyserial
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_x86_64.whl (13 kB)
Collecting tenacity
  Using cached tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting ntc-templates
  Using cached ntc_templates-3.0.0-py3-none-any.whl (303 kB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-3.2.2-cp36-abi3-macosx_10_10_universal2.whl (50 kB)
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.5.0-cp36-abi3-macosx_10_10_universal2.whl (349 kB)
Collecting cryptography>=2.5
  Using cached cryptography-37.0.4-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.0-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.9/139.9 kB 7.7 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting toml==0.10.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting passlib<2.0.0,>=1.7.4
  Using cached passlib-1.7.4-py2.py3-none-any.whl (525 kB)
Collecting dnspython<3.0.0,>=2.1.0
  Using cached dnspython-2.2.1-py3-none-any.whl (269 kB)
Collecting loguru==0.6.0
  Using cached loguru-0.6.0-py3-none-any.whl (58 kB)
Installing collected packages: pyserial, passlib, netaddr, librouteros, urllib3, toml, tenacity, six, pyYAML, pyparsing, pyeapi, pycparser, MarkupSafe, lxml, loguru, idna, future, dnspython, charset-normalizer, certifi, yamlordereddictloader, transitions, textfsm, requests, jinja2, ciscoconfparse, cffi, pynacl, ntc-templates, cryptography, bcrypt, paramiko, scp, ncclient, netmiko, junos-eznc, napalm, napalm-ros
Successfully installed MarkupSafe-2.1.1 bcrypt-3.2.2 certifi-2022.6.15 cffi-1.15.1 charset-normalizer-2.1.0 ciscoconfparse-1.6.40 cryptography-37.0.4 dnspython-2.2.1 future-0.18.2 idna-3.3 jinja2-3.1.2 junos-eznc-2.6.5 librouteros-3.2.1 loguru-0.6.0 lxml-4.9.1 napalm-3.4.1 napalm-ros-1.0.1 ncclient-0.6.13 netaddr-0.8.0 netmiko-3.4.0 ntc-templates-3.0.0 paramiko-2.11.0 passlib-1.7.4 pyYAML-6.0 pycparser-2.21 pyeapi-0.8.4 pynacl-1.5.0 pyparsing-3.0.9 pyserial-3.5 requests-2.28.1 scp-0.14.4 six-1.16.0 tenacity-8.0.1 textfsm-1.1.3 toml-0.10.2 transitions-0.8.11 urllib3-1.26.11 yamlordereddictloader-0.4.0

[notice] A new release of pip available: 22.1.2 -> 22.2.1
[notice] To update, run: pip install --upgrade pip
(venv) MacBook-Pro-5:test1 $ pip freeze | egrep 'napalm|miko'
napalm==3.4.1
napalm-ros==1.0.1
netmiko==3.4.0
paramiko==2.11.0
(venv) MacBook-Pro-5:test1 $

And here is pip install napalm napalm-ros:

MacBook-Pro-5:tmp $ mkdir test2
MacBook-Pro-5:tmp $ cd test2
MacBook-Pro-5:test2 $ virtualenv venv
created virtual environment CPython3.10.4.final.0-64 in 613ms
  creator CPython3Posix(dest=/Users/brian/tmp/test2/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/brian/Library/Application Support/virtualenv)
    added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
MacBook-Pro-5:test2 $ . venv/bin/activate
(venv) MacBook-Pro-5:test2 $ pip install napalm napalm-ros
Collecting napalm
  Using cached napalm-4.0.0-py2.py3-none-any.whl (266 kB)
Collecting napalm-ros
  Using cached napalm_ros-1.0.1-py3-none-any.whl
Collecting scp
  Using cached scp-0.14.4-py2.py3-none-any.whl (8.6 kB)
Collecting junos-eznc>=2.6.3
  Using cached junos_eznc-2.6.5-py2.py3-none-any.whl (197 kB)
Collecting textfsm<=1.1.2
  Using cached textfsm-1.1.2-py2.py3-none-any.whl (44 kB)
Collecting typing-extensions>=4.3.0
  Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Collecting paramiko>=2.6.0
  Using cached paramiko-2.11.0-py2.py3-none-any.whl (212 kB)
Collecting ttp-templates
  Using cached ttp_templates-0.3.1-py3-none-any.whl (61 kB)
Collecting jinja2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting pyYAML
  Using cached PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl (197 kB)
Collecting lxml>=4.3.0
  Using cached lxml-4.9.1-cp310-cp310-macosx_12_0_x86_64.whl
Collecting ncclient
  Using cached ncclient-0.6.13-py2.py3-none-any.whl
Collecting requests>=2.7.0
  Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting cffi>=1.11.3
  Using cached cffi-1.15.1-cp310-cp310-macosx_10_9_x86_64.whl (179 kB)
Collecting netutils>=1.0.0
  Using cached netutils-1.1.0-py3-none-any.whl (111 kB)
Collecting future
  Using cached future-0.18.2-py3-none-any.whl
Collecting netaddr
  Using cached netaddr-0.8.0-py2.py3-none-any.whl (1.9 MB)
Collecting netmiko>=4.0.0
  Using cached netmiko-4.1.1-py3-none-any.whl (192 kB)
Requirement already satisfied: setuptools>=38.4.0 in ./venv/lib/python3.10/site-packages (from napalm) (62.6.0)
Collecting ttp
  Using cached ttp-0.9.1-py2.py3-none-any.whl (84 kB)
Collecting pyeapi>=0.8.2
  Using cached pyeapi-0.8.4-py3-none-any.whl
Collecting librouteros==3.*
  Using cached librouteros-3.2.1-py3-none-any.whl
Collecting napalm
  Using cached napalm-3.4.1-py2.py3-none-any.whl (256 kB)
  Using cached napalm-3.4.0-py2.py3-none-any.whl (256 kB)
  Using cached napalm-3.3.1-py2.py3-none-any.whl (256 kB)
Collecting ciscoconfparse
  Using cached ciscoconfparse-1.6.40-py3-none-any.whl (136 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting pyserial
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyparsing
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting transitions
  Using cached transitions-0.8.11-py2.py3-none-any.whl (83 kB)
Collecting yamlordereddictloader
  Using cached yamlordereddictloader-0.4.0-py3-none-any.whl
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_x86_64.whl (13 kB)
Collecting ntc-templates>=2.0.0
  Using cached ntc_templates-3.0.0-py3-none-any.whl (303 kB)
Collecting tenacity
  Using cached tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.5.0-cp36-abi3-macosx_10_10_universal2.whl (349 kB)
Collecting cryptography>=2.5
  Using cached cryptography-37.0.4-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-3.2.2-cp36-abi3-macosx_10_10_universal2.whl (50 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.6.15-py3-none-any.whl (160 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.0-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
Collecting passlib<2.0.0,>=1.7.4
  Using cached passlib-1.7.4-py2.py3-none-any.whl (525 kB)
Collecting loguru==0.6.0
  Using cached loguru-0.6.0-py3-none-any.whl (58 kB)
Collecting toml==0.10.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting dnspython<3.0.0,>=2.1.0
  Using cached dnspython-2.2.1-py3-none-any.whl (269 kB)
Installing collected packages: pyserial, passlib, netaddr, librouteros, urllib3, toml, tenacity, six, pyYAML, pyparsing, pyeapi, pycparser, MarkupSafe, lxml, loguru, idna, future, dnspython, charset-normalizer, certifi, yamlordereddictloader, transitions, textfsm, requests, jinja2, ciscoconfparse, cffi, pynacl, ntc-templates, cryptography, bcrypt, paramiko, scp, ncclient, netmiko, junos-eznc, napalm, napalm-ros
Successfully installed MarkupSafe-2.1.1 bcrypt-3.2.2 certifi-2022.6.15 cffi-1.15.1 charset-normalizer-2.1.0 ciscoconfparse-1.6.40 cryptography-37.0.4 dnspython-2.2.1 future-0.18.2 idna-3.3 jinja2-3.1.2 junos-eznc-2.6.5 librouteros-3.2.1 loguru-0.6.0 lxml-4.9.1 napalm-3.3.1 napalm-ros-1.0.1 ncclient-0.6.13 netaddr-0.8.0 netmiko-4.1.1 ntc-templates-3.0.0 paramiko-2.11.0 passlib-1.7.4 pyYAML-6.0 pycparser-2.21 pyeapi-0.8.4 pynacl-1.5.0 pyparsing-3.0.9 pyserial-3.5 requests-2.28.1 scp-0.14.4 six-1.16.0 tenacity-8.0.1 textfsm-1.1.2 toml-0.10.2 transitions-0.8.11 urllib3-1.26.11 yamlordereddictloader-0.4.0

[notice] A new release of pip available: 22.1.2 -> 22.2.1
[notice] To update, run: pip install --upgrade pip
(venv) MacBook-Pro-5:test2 $ pip freeze | egrep 'napalm|miko'
napalm==3.3.1
napalm-ros==1.0.1
netmiko==4.1.1
paramiko==2.11.0
(venv) MacBook-Pro-5:test2 $

Łukasz Kostka · Answer 6 · Sat Jul 30 2022 02:57:22 GMT+0800 (China Standard Time)

In one install you get:

napalm==3.4.1

and in another:

napalm==3.3.1

Any way, napalm-ros will work now with napalm==4.*. I've released 1.1.0

Brian Candler · Answer 7 · Sat Jul 30 2022 03:02:51 GMT+0800 (China Standard Time)

Sorry, I didn't make the point clear.

In the first install, I get netmiko-3.4.0 and all is good. In the second I get netmiko-4.1.1, and napalm 3.3.1 is incompatible with netmiko 4.x.

>>> import napalm
...
ModuleNotFoundError: No module named 'netmiko.ssh_exception'

Łukasz Kostka · Answer 8 · Sat Jul 30 2022 06:59:43 GMT+0800 (China Standard Time)

Use version pinning.