RoadRecon breaks due to SQLAlchemy dependency
Paradoxis opened this issue · comments
A recent change in SQLAlchemy appears to have broken roadrecon. When running the tool with no dependencies, the following stack trace is printed:
$ roadrecon
Traceback (most recent call last):
File "/home/kali/.local/bin/roadrecon", line 5, in <module>
from roadtools.roadrecon.main import main
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.8/site-packages/roadtools/roadrecon/main.py", line 6, in <module>
from roadtools.roadrecon.gather import getargs as getgatherargs
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.8/site-packages/roadtools/roadrecon/gather.py", line 14, in <module>
from roadtools.roadlib.metadef.database import User, ServicePrincipal, Application, Group, Device, DirectoryRole, RoleAssignment, ExtensionProperty, Contact, OAuth2PermissionGrant, Policy, RoleDefinition, AppRoleAssignment, TenantDetail
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.8/site-packages/roadtools/roadlib/metadef/database.py", line 5, in <module>
from sqlalchemy import Column, Text, Boolean, BigInteger as Integer, Binary, create_engine, Table, ForeignKey
ImportError: cannot import name 'Binary' from 'sqlalchemy' (/home/kali/.local/pipx/venvs/roadrecon/lib/python3.8/site-packages/sqlalchemy/__init__.py)
The package was installed like so:
$ pipx install roadrecon
The issue is present on both Python 3.8, as well as Python 3.10. I also attempted to downgrade SQLAchemy to 1.4.40, 1.4.35 and 1.3.24 which did not work.
The current roadrecon version appears to ship with an older version of roadlib, when updating it forcibly inside of the virutalenv the tool still fails to run (although due to another error):
┌──(kali㉿kali)-[~]
└─$ roadrecon
Traceback (most recent call last):
File "/home/kali/.local/bin/roadrecon", line 8, in <module>
sys.exit(main())
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/main.py", line 85, in main
plugin_module = importlib.import_module('roadtools.roadrecon.plugins.{}'.format(plugin))
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/plugins/xlsexport.py", line 43, in <module>
from roadtools.roadrecon.server import (
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/server.py", line 2, in <module>
from flask_sqlalchemy import SQLAlchemy
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/__init__.py", line 5, in <module>
from .extension import SQLAlchemy
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/extension.py", line 17, in <module>
from .model import _QueryProperty
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/model.py", line 210, in <module>
class DefaultMeta(BindMetaMixin, NameMetaMixin, sa.orm.DeclarativeMeta):
AttributeError: module 'sqlalchemy.orm' has no attribute 'DeclarativeMeta'
Steps taken:
┌──(kali㉿kali)-[~]
└─$ pipx install roadrecon
installed package roadrecon 0.11.0, installed using Python 3.10.7
These apps are now globally available
- roadrecon
- roadrecon-gui
done! ✨ 🌟 ✨
┌──(kali㉿kali)-[~]
└─$ /home/kali/.local/pipx/venvs/roadrecon/bin/python -m pip freeze
adal==1.2.7
aiohttp==3.8.3
aiosignal==1.2.0
async-timeout==4.0.2
attrs==22.1.0
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
cryptography==38.0.1
et-xmlfile==1.1.0
Flask==2.2.2
Flask-Cors==3.0.10
flask-marshmallow==0.14.0
Flask-SQLAlchemy==3.0.0
frozenlist==1.3.1
greenlet==1.1.3
idna==3.4
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
marshmallow==3.18.0
marshmallow-sqlalchemy==0.28.1
multidict==6.0.2
openpyxl==3.0.10
packaging==21.3
pycparser==2.21
PyJWT==1.7.1
pyparsing==3.0.9
python-dateutil==2.8.2
requests==2.28.1
roadlib==0.11.1
roadrecon==0.11.0
six==1.16.0
SQLAlchemy==1.4.41
urllib3==1.26.12
Werkzeug==2.2.2
yarl==1.8.1
┌──(kali㉿kali)-[~]
└─$ /home/kali/.local/pipx/venvs/roadrecon/bin/python -m pip install -U roadlib
Requirement already satisfied: roadlib in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (0.11.1)
Collecting roadlib
Using cached roadlib-0.12.0-py3-none-any.whl (22 kB)
Requirement already satisfied: pyjwt<2.0 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from roadlib) (1.7.1)
Requirement already satisfied: adal in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from roadlib) (1.2.7)
Collecting sqlalchemy<1.4
Using cached SQLAlchemy-1.3.24-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: requests<3,>=2.0.0 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from adal->roadlib) (2.28.1)
Requirement already satisfied: python-dateutil<3,>=2.1.0 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from adal->roadlib) (2.8.2)
Requirement already satisfied: cryptography>=1.1.0 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from adal->roadlib) (38.0.1)
Requirement already satisfied: cffi>=1.12 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from cryptography>=1.1.0->adal->roadlib) (1.15.1)
Requirement already satisfied: six>=1.5 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from python-dateutil<3,>=2.1.0->adal->roadlib) (1.16.0)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from requests<3,>=2.0.0->adal->roadlib) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from requests<3,>=2.0.0->adal->roadlib) (2022.9.24)
Requirement already satisfied: idna<4,>=2.5 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from requests<3,>=2.0.0->adal->roadlib) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from requests<3,>=2.0.0->adal->roadlib) (1.26.12)
Requirement already satisfied: pycparser in ./.local/pipx/venvs/roadrecon/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=1.1.0->adal->roadlib) (2.21)
Installing collected packages: sqlalchemy, roadlib
Attempting uninstall: sqlalchemy
Found existing installation: SQLAlchemy 1.4.41
Uninstalling SQLAlchemy-1.4.41:
Successfully uninstalled SQLAlchemy-1.4.41
Attempting uninstall: roadlib
Found existing installation: roadlib 0.11.1
Uninstalling roadlib-0.11.1:
Successfully uninstalled roadlib-0.11.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flask-sqlalchemy 3.0.0 requires SQLAlchemy>=1.4.18, but you have sqlalchemy 1.3.24 which is incompatible.
Successfully installed roadlib-0.12.0 sqlalchemy-1.3.24
┌──(kali㉿kali)-[~]
└─$ roadrecon
Traceback (most recent call last):
File "/home/kali/.local/bin/roadrecon", line 8, in <module>
sys.exit(main())
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/main.py", line 85, in main
plugin_module = importlib.import_module('roadtools.roadrecon.plugins.{}'.format(plugin))
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/plugins/xlsexport.py", line 43, in <module>
from roadtools.roadrecon.server import (
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/roadtools/roadrecon/server.py", line 2, in <module>
from flask_sqlalchemy import SQLAlchemy
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/__init__.py", line 5, in <module>
from .extension import SQLAlchemy
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/extension.py", line 17, in <module>
from .model import _QueryProperty
File "/home/kali/.local/pipx/venvs/roadrecon/lib/python3.10/site-packages/flask_sqlalchemy/model.py", line 210, in <module>
class DefaultMeta(BindMetaMixin, NameMetaMixin, sa.orm.DeclarativeMeta):
AttributeError: module 'sqlalchemy.orm' has no attribute 'DeclarativeMeta'
All right, I think i've figured out what was going wrong here. The latest version of flask-sqlalchemy forces version 1.4 of sqlalchemy. The latest verion of roadlib that was on pypi at the time of writing, 0.12.0, did not support that version. Version 0.11.1 did not support that version either, but did not have that explicitly in the requirements, making pip think that it supports the latest sqlalchemy just fine and thus installing that version. Of course now you ended up with both an old roadlib version and with an incompatible sqlalchemy.
I actually removed the sqlalchemy requirements in version 0.12.1 of roadlib, but apparently forgot to upload that version to pypi, so the old 0.12.0 was still the latest.
I have now uploaded roadlib 0.13.0 to pypi, which works with the latest sqlalchemy version without errors. Let me know if this solves the installation issues for you.
Appears to be working again on my end :)