Incorrect bytecode and envier dependencies

Question

Incorrect bytecode and envier dependencies

unpollito opened this issue 3 months ago · comments

David Martín-González commented 3 months ago

Summary of problem

After upgrading ddtrace from 1.12.2 to 2.7.2, tracing for some modules (e.g., Django and Redis stopped working). Turns out there was a crash because my version of bytecode was still 0.13.0, but ddtrace is invoking bc.TryBegin, which seems to have been added on version 0.14.0. Pinning bytecode to use the latest version available at the time of writing (0.15.1) fixed that issue for me.

However, another problem appeared, this time with envier (TypeError: Env.v() got an unexpected keyword argument 'private'). I had version 0.4.0, but updating it to 0.5.1 fixed the issue. I haven't been able to find a changelog for this package, so I don't know which is the version where the relevant change was introduced.

I would suggest adding explicit dependencies on the relevant versions, so that other people upgrading to more recent versions of ddtrace don't come across the same issues.

Which version of dd-trace-py are you using?

2.7.2

Which version of pip are you using?

24.0

Which libraries and their versions are you using?

This is before any of the updates:

aiohttp==3.9.3
aiosignal==1.3.1
alembic==1.8.1
algoliasearch==3.0.0
amqp==5.2.0
analytics-python==1.4.0
argh==0.26.2
asgiref==3.5.2
astroid==3.1.0
asttokens==2.0.8
async-timeout==4.0.2
attrs==22.1.0
auth0-python==3.24.0
autoflake==1.7.5
autopep8==1.7.0
Babel==2.11.0
backcall==0.2.0
backoff==1.10.0
base-repr==1.0.6
billiard==4.2.0
black==22.8.0
bytecode==0.15.1
CacheControl==0.12.14
cachetools==4.2.4
cachy==0.3.0
cattrs==22.2.0
celery==5.3.6
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
charset-normalizer==2.1.1
cleo==1.0.0a5
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
coverage==4.5.4
crashtest==0.3.1
cron-descriptor==1.4.3
cryptography==38.0.3
datadog==0.44.0
dateparser==1.1.4
ddsketch==2.0.4
ddtrace==2.7.2
decorator==5.1.1
Deprecated==1.2.13
dill==0.3.8
distlib==0.3.6
Django==3.2.18
django-appconf==1.0.5
django-cache-memoize==0.1.10
django-celery-beat==2.5.0
django-celery-results==2.0.1
django-cors-headers==3.13.0
django-cryptography==1.1
django-db-geventpool==4.0.1
django-debug-toolbar==3.5.0
django-filter==22.1
django-hashid-field==3.3.7
django-request-logging==0.7.5
django-silk==5.0.1
django-timezone-field==6.1.0
djangorestframework==3.14.0
djangorestframework-bulk==0.2.1
dulwich==0.20.50
envier==0.4.0
execnet==1.9.0
executing==1.0.0
expiringdict==1.2.2
factory-boy==3.2.1
Faker==15.3.4
filelock==3.8.0
filestack-python==3.5.0
freezegun==1.2.2
frozenlist==1.4.0
gevent==22.10.2
google-api-core==1.33.0
google-api-python-client==2.68.0
google-auth==2.14.1
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.3
google-cloud-core==0.29.1
google-cloud-logging==1.10.0
google-cloud-sqlcommenter==2.0.0
googleapis-common-protos==1.56.4
gprof2dot==2022.7.29
greenlet==3.0.3
grpcio==1.62.1
grpcio-status==1.48.1
gspread==5.7.2
gunicorn==21.2.0
hashids==1.3.1
html5lib==1.1
httplib2==0.20.4
identify==2.5.5
idna==3.3
ijson==3.2.0.post0
importlib-metadata==6.0.1
iniconfig==2.0.0
ipdb==0.13.9
ipython==8.5.0
isort==5.12.0
jaraco.classes==3.3.1
jedi==0.18.1
jeepney==0.8.0
Jinja2==3.1.2
jsonpath-ng==1.5.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
keyring==24.3.1
kombu==5.3.5
launchdarkly-server-sdk==8.1.1
lockfile==0.12.2
lz4==4.3.2
Mako==1.2.2
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
mccabe==0.7.0
mixpanel==4.10.0
monotonic==1.6
more-itertools==10.2.0
msgpack==1.0.4
multidict==6.0.4
mypy-extensions==0.4.3
ndg-httpsclient==0.5.1
networkx==3.0
nodeenv==1.7.0
numpy==1.26.4
oauth2client==2.0.1
oauthlib==3.2.1
openai==0.28.1
opentelemetry-api==1.17.0
packaging==21.3
pandas==2.2.1
parso==0.8.3
pathspec==0.10.1
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.10.0
platformdirs==2.5.2
pluggy==1.0.0
ply==3.11
poetry==1.2.2
poetry-core==1.3.2
poetry-plugin-export==1.2.0
pre-commit==2.20.0
prompt-toolkit==3.0.43
protobuf==3.20.1
psycogreen==1.0.2
psycopg2==2.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pusher==3.3.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.9.1
pycparser==2.21
pyflakes==2.5.0
Pygments==2.13.0
PyJWT==1.7.1
pylev==1.4.0
pylint==3.1.0
pymemcache==4.0.0
PyNaCl==1.5.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyre2==0.3.6
pyRFC3339==1.1
pytest==7.3.1
pytest-django==4.5.2
pytest-xdist==3.3.1
python-crontab==2.6.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-http-client==3.3.7
python-json-logger==2.0.4
python-memcached==1.59
python-slugify==7.0.0
pytz==2022.2.1
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0.1
redis==4.3.4
referencing==0.33.0
regex==2022.3.2
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
responses==0.21.0
rpds-py==0.18.0
rsa==4.9
SecretStorage==3.3.3
semver==2.13.0
sendgrid==5.2.0
sentry-sdk==1.41.0
setuptools==65.3.0
shellingham==1.5.4
shortuuid==1.0.11
six==1.16.0
slack-sdk==3.21.3
SQLAlchemy==1.3.24
sqlparse==0.4.2
stack-data==0.5.0
stripe==4.2.0
tabulate==0.9.0
tblib==1.7.0
text-unidecode==1.3
toml==0.10.2
tomlkit==0.11.6
tqdm==4.64.1
trafaret==2.0.2
traitlets==5.4.0
typing_extensions==4.10.0
tzdata==2024.1
tzlocal==4.2
uritemplate==3.0.1
urllib3==1.26.12
vine==5.1.0
virtualenv==20.16.5
watchdog==2.1.9
wcwidth==0.2.5
webencodings==0.5.1
wheel==0.43.0
wrapt==1.14.1
xmltodict==0.13.0
yarl==1.9.2
zipp==3.15.0
zope.event==4.5.0
zope.interface==5.4.0

How can we reproduce your problem?

Installing the aforementioned versions of the libraries should be enough to reproduce this.

What is the result that you get?

Before pinning the bytecode version:

error configuring Datadog tracing
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/sitecustomize.py", line 116, in <module>
    import ddtrace.bootstrap.preload as preload  # Perform the actual initialisation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/preload.py", line 8, in <module>
    from ddtrace.debugging._config import di_config  # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/debugging/__init__.py", line 31, in <module>
    from ddtrace.debugging._debugger import Debugger as DynamicInstrumentation
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/debugging/_debugger.py", line 32, in <module>
    from ddtrace.debugging._function.store import FullyNamedWrappedFunction
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/debugging/_function/store.py", line 12, in <module>
    from ddtrace.internal.injection import HookInfoType
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/injection.py", line 11, in <module>
    from ddtrace.internal.assembly import Assembly
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/assembly.py", line 25, in <module>
    ParsedLine = t.Union[bc.Instr, bc.Label, bc.TryBegin, bc.TryEnd]
                                             ^^^^^^^^^^^
AttributeError: module 'bytecode' has no attribute 'TryBegin'

This while starting the app, and lost traces for the services that get initialized dynamically.

Before pinning the Envier version:

error configuring Datadog tracing
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/sitecustomize.py", line 116, in <module>
    import ddtrace.bootstrap.preload as preload  # Perform the actual initialisation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/preload.py", line 18, in <module>
    from ddtrace.settings.symbol_db import config as symdb_config  # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/settings/symbol_db.py", line 6, in <module>
    class SymbolDatabaseConfig(En):
  File "/usr/local/lib/python3.12/site-packages/ddtrace/settings/symbol_db.py", line 28, in SymbolDatabaseConfig
    _force = En.v(
             ^^^^^
TypeError: Env.v() got an unexpected keyword argument 'private'
Using cache 'memcached': {'default': {'BACKEND': 'app.cache.RequestAwareCache', 'LOCATION': 'unique-snowflake'}, 'primary_cache': {'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', 'KEY_FUNCTION': 'app.settings.make_key', 'LOCATION': 'memcached:11211'}, 'local': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', 'LOCATION': 'unique-snowflake'}}
error configuring Datadog tracing
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/sitecustomize.py", line 116, in <module>
    import ddtrace.bootstrap.preload as preload  # Perform the actual initialisation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/bootstrap/preload.py", line 18, in <module>
    from ddtrace.settings.symbol_db import config as symdb_config  # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/usr/local/lib/python3.12/site-packages/ddtrace/internal/module.py", line 208, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.12/site-packages/ddtrace/settings/symbol_db.py", line 6, in <module>
    class SymbolDatabaseConfig(En):
  File "/usr/local/lib/python3.12/site-packages/ddtrace/settings/symbol_db.py", line 28, in SymbolDatabaseConfig
    _force = En.v(
             ^^^^^
TypeError: Env.v() got an unexpected keyword argument 'private'

What is the result that you expected?

No crash, and traces for dynamically initialized services are sent.

Gabriele N. Tornetta · Answer 1 · Fri Mar 15 2024 00:50:32 GMT+0800 (China Standard Time)

@unpollito thanks for the report. We have already fixed this on the main branch, and we have backports ready to be merged (e.g. #8639).

Munir Abdinur · Answer 2 · Sat Mar 16 2024 05:12:31 GMT+0800 (China Standard Time)

This issue should be resolved in v2.7.3: https://github.com/DataDog/dd-trace-py/releases/tag/v2.7.3