[Bug] Dependency using a deprecated and removed module (`pkg_resources`)

Question

[Bug] Dependency using a deprecated and removed module (`pkg_resources`)

brokensound77 opened this issue 3 months ago · comments

Describe the Bug

The dependency marshmallow_jsonschema depends on pkg_resources, which has been deprecated and removed as of python 3.12. This is especially problematic since we enforce py312 in this repo.

An issue exists in the upstream library to update, but no progress has been made. The last update was over a year ago, so unsure if it is being maintained.

Per https://docs.python.org/3/whatsnew/3.12.html:

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

pkg_resources was moved to within setuptools, so the simplest fix would be to add setuptools as a required dependency

Although it is included in the Makefile to install setuptools as a prepatory step, this resolves the issue when make is unavailable or when this is being installed and imported as a dependency.

Full error (occurring on execution not install):

import detection_rules

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/__init__.py", line 13, in <module>
    from . import (  # noqa: E402
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/devtools.py", line 32, in <module>
    from . import attack, rule_loader, utils
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/rule_loader.py", line 20, in <module>
    from .mappings import RtaMappings
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/mappings.py", line 12, in <module>
    from .rule import TOMLRule
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/rule.py", line 33, in <module>
    from .mixins import MarshmallowDataclassMixin, StackCompatMixin
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/mixins.py", line 15, in <module>
    import marshmallow_jsonschema
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/marshmallow_jsonschema/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'

IINM, it is also the actual fix to the distutils problem outline in troubleshooting.md

The best fix for both of these would be to remove our dependency on the deprecated libraries, but that would require upstream lib changes.

From the pkg_resources docs:

Use of pkg_resources is deprecated in favor of importlib.resources, importlib.metadata and their backports (importlib_resources, importlib_metadata). Some useful APIs are also provided by packaging (e.g. requirements and version parsing). Users should refrain from new usage of pkg_resources and should work to port to importlib-based solutions.

To Reproduce

pip install .
import detection_rules

Expected Behavior

No response

Screenshots

No response

Desktop - OS

None

Desktop - Version

No response

Additional Context

No response

shashank-elastic · Answer 1 · Wed Sep 18 2024 21:36:30 GMT+0800 (China Standard Time)

The error is reproducible on pip install .

Created a new venv

detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  shashank.suryanarayana@elastic.co 
❯ python3 -m venv .venv_new    
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  shashank.suryanarayana@elastic.co 
❯ source .venv_new/bin/activate
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv_new) on ☁️  shashank.suryanarayana@elastic.co 
❯

pip install

❯ pip install .
Looking in indexes: https://pypi.org/simple, https://shashank.suryanarayana%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing /Users/shashankks/elastic_workspace/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/jk/t_tlgnwx4w998xqw3_kjzyx00000gn/T/pip-install-s6kwdf_x/detection-rules-kql_23861e9674b34fbcbe7eb5703d2e834c
Successfully built detection_rules detection-rules-kibana detection-rules-kql
Installing collected packages: pytoml, lark-parser, jsl, XlsxWriter, urllib3, typing-extensions, typeguard, toml, semver, rpds-py, PyYAML, packaging, mypy-extensions, marko, idna, eql, Click, charset-normalizer, certifi, attrs, typing-inspect, requests, referencing, marshmallow, elastic-transport, detection-rules-kql, marshmallow-union, marshmallow-jsonschema, marshmallow-dataclass, jsonschema-specifications, elasticsearch, jsonschema, detection-rules-kibana, detection_rules
Successfully installed Click-8.1.7 PyYAML-6.0.2 XlsxWriter-3.2.0 attrs-24.2.0 certifi-2024.8.30 charset-normalizer-3.3.2 detection-rules-kibana-0.4.0 detection-rules-kql-0.1.7 detection_rules-0.1.0 elastic-transport-8.15.0 elasticsearch-8.12.1 eql-0.9.19 idna-3.10 jsl-0.2.4 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 lark-parser-0.12.0 marko-2.0.3 marshmallow-3.21.3 marshmallow-dataclass-8.6.1 marshmallow-jsonschema-0.13.0 marshmallow-union-0.1.15.post1 mypy-extensions-1.0.0 packaging-24.1 pytoml-0.1.21 referencing-0.35.1 requests-2.31.0 rpds-py-0.20.0 semver-3.0.2 toml-0.10.2 typeguard-3.0.2 typing-extensions-4.10.0 typing-inspect-0.9.0 urllib3-2.2.3
(.venv)

Error

    import marshmallow_jsonschema
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/Users/shashankks/elastic_workspace/detection-rules/.venv_new/lib/python3.12/site-packages/marshmallow_jsonschema/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pkg_resources'
>>>

Also noticed we don't notice this error on pip install ".[dev]"

Created a new venv

❯ python3 -m venv .venv_new1   
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  shashank.suryanarayana@elastic.co 
❯ source .venv_new1/bin/activate
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv_new1) on ☁️  shashank.suryanarayana@elastic.co 
❯

pip install dev

❯ pip install ".[dev]"
Looking in indexes: https://pypi.org/simple, https://shashank.suryanarayana%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing /Users/shashankks/elastic_workspace/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/jk/t_tlgnwx4w998xqw3_kjzyx00000gn/T/pip-install-8kydmjrf/detection-rules-kql_fb529215442b43a8b907f5de29561a96
Successfully installed Click-8.1.7 Deprecated-1.2.14 PyGithub-2.2.0 PyYAML-6.0.2 XlsxWriter-3.2.0 attrs-24.2.0 certifi-2024.8.30 cffi-1.17.1 cfgv-3.4.0 charset-normalizer-3.3.2 cryptography-43.0.1 detection-rules-kibana-0.4.0 detection-rules-kql-0.1.7 detection_rules-0.1.0 distlib-0.3.8 elastic-transport-8.15.0 elasticsearch-8.12.1 eql-0.9.19 filelock-3.16.1 flake8-7.0.0 identify-2.6.1 idna-3.10 iniconfig-2.0.0 jsl-0.2.4 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 lark-parser-0.12.0 marko-2.0.3 marshmallow-3.21.3 marshmallow-dataclass-8.6.1 marshmallow-jsonschema-0.13.0 marshmallow-union-0.1.15.post1 mccabe-0.7.0 mypy-extensions-1.0.0 nodeenv-1.8.0 packaging-24.1 pep8-naming-0.13.0 platformdirs-4.3.6 pluggy-1.5.0 pre-commit-3.6.2 pycodestyle-2.11.1 pycparser-2.22 pyflakes-3.2.0 pyjwt-2.9.0 pynacl-1.5.0 pytest-8.3.3 pytoml-0.1.21 referencing-0.35.1 requests-2.31.0 rpds-py-0.20.0 semver-3.0.2 setuptools-75.1.0 toml-0.10.2 typeguard-3.0.2 typing-extensions-4.10.0 typing-inspect-0.9.0 urllib3-2.2.3 virtualenv-20.26.5 wrapt-1.16.0
(.venv)

Note setuptools-75.1.0 being installed as part of dev

No Error

# code object from '/Users/shashankks/elastic_workspace/detection-rules/detection_rules/__pycache__/ml.cpython-312.pyc'
import 'detection_rules.ml' # <_frozen_importlib_external.SourceFileLoader object at 0x10e348bf0>
import 'detection_rules' # <_frozen_importlib_external.SourceFileLoader object at 0x1005bf350>
>>>

We can freeze setuptools==75.1.0 as a dependency non dev installs for pip @Mikaayenson

But if we want to remove our dependancy on pkg_resources, this would take some more cycles to discuss do we wanna do that , and how do we wanna do that.

Justin Ibarra · Answer 2 · Fri Sep 27 2024 00:16:13 GMT+0800 (China Standard Time)

Just freeze setuptools to the latest and add that as a dependency under pyproject. Then you can also simplify the makefile be removing explicit install.

Also, if it fully resolves the disutils issue, you could also delete this troubleshooting guide - but you would need to test under those exact circumstances to validate

Sergey Polzunov · Answer 3 · Wed Oct 09 2024 21:18:34 GMT+0800 (China Standard Time)

getting the same error running make on a clean system:

...
Successfully built detection_rules
Installing collected packages: detection_rules
  Attempting uninstall: detection_rules
    Found existing installation: detection_rules 0.1.0
    Uninstalling detection_rules-0.1.0:
      Successfully uninstalled detection_rules-0.1.0
Successfully installed detection_rules-0.1.0

RELEASE:
./env/detection-rules-build/bin/python -m detection_rules dev build-release --generate-navigator
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/Users/traut/Work/detection-rules/detection_rules/__init__.py", line 13, in <module>
    from . import (  # noqa: E402
  File "/Users/traut/Work/detection-rules/detection_rules/custom_rules.py", line 12, in <module>
    from .main import root
  File "/Users/traut/Work/detection-rules/detection_rules/main.py", line 22, in <module>
    from .action_connector import (TOMLActionConnectorContents,
  File "/Users/traut/Work/detection-rules/detection_rules/action_connector.py", line 15, in <module>
    from .mixins import MarshmallowDataclassMixin
  File "/Users/traut/Work/detection-rules/detection_rules/mixins.py", line 15, in <module>
    import marshmallow_jsonschema
  File "/Users/traut/Work/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_jsonschema/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution
ModuleNotFoundError: No module named 'pkg_resources'
make: *** [release] Error 1

the fact that the script uses internal venv (under ./env directory) makes it a bit confusing but:

global env has setuptools installed: setuptools==75.1.0

venv in ./env does not:

$ . ./env/detection-rules-build/bin/activate
(detection-rules-build) $ pip freeze | grep setuptools
(detection-rules-build) $

Mika Ayenson · Answer 4 · Wed Oct 09 2024 23:37:01 GMT+0800 (China Standard Time)

FYI I haven't fully investigated all the places where pkg_resourcesis pulled in, but it appears that one is withmarshmallow_jsonschema. It appears that we only use this in some logic within mixins` written 3 years ago. Additionally that package isn't actively maintained. I simply removed the code (assuming the core libraries have evolved over the last couple years), and reran the unit tests. It appears the unit tests at least still pass.

diff --git a/detection_rules/mixins.py b/detection_rules/mixins.py
index b22677d29..c31bbb3fb 100644
--- a/detection_rules/mixins.py
+++ b/detection_rules/mixins.py
@@ -12,11 +12,11 @@ from typing import Any, Optional, TypeVar, Type, Literal
 import json
 import marshmallow_dataclass
 import marshmallow_dataclass.union_field
-import marshmallow_jsonschema
+# import marshmallow_jsonschema
 import marshmallow_union
 import marshmallow
 from marshmallow import Schema, ValidationError, validates_schema, fields as marshmallow_fields
-
+from jsonschema import Draft7Validator
 from .config import load_current_package_version
 from .schemas import definitions
 from .schemas.stack_compat import get_incompatible_fields
@@ -127,13 +127,38 @@ class MarshmallowDataclassMixin:
         """Get a key from the query data without raising attribute errors."""
         return getattr(self, key, default)
 
+    # @classmethod
+    # @cached
+    # def jsonschema(cls):
+    #     """Get the jsonschema representation for this class."""
+    #     jsonschema = PatchedJSONSchema().dump(cls.__schema())
+    #     jsonschema = patch_jsonschema(jsonschema)
+    #     return jsonschema
+    # @classmethod
+    # @cached
+    # def jsonschema(cls):
+    #     """Get the jsonschema representation for this class."""
+    #     # Directly use marshmallow_dataclass and jsonschema
+    #     schema = cls.__schema()
+    #     schema_dict = schema.dump(cls)  # Get the schema as a dictionary
+
+    #     # Patch the schema if necessary (assuming patch_jsonschema is still useful)
+    #     return patch_jsonschema(schema_dict)
+
     @classmethod
     @cached
     def jsonschema(cls):
         """Get the jsonschema representation for this class."""
-        jsonschema = PatchedJSONSchema().dump(cls.__schema())
-        jsonschema = patch_jsonschema(jsonschema)
-        return jsonschema
+        schema = cls.__schema()
+        schema_dict = schema.dump(cls)  # Get the schema as a dictionary
+
+        # Validate the schema itself
+        Draft7Validator.check_schema(schema_dict)  # This will raise an error if the schema is invalid
+
+        # Optionally patch the schema if you still need to modify it
+        schema_dict = patch_jsonschema(schema_dict)
+
+        return schema_dict
 
     @classmethod
     def from_dict(cls: Type[ClassT], obj: dict, unknown: Optional[UNKNOWN_VALUES] = None) -> ClassT:
@@ -227,25 +252,25 @@ class StackCompatMixin:
                                       f'min compatibility: {min_compat}, max compatibility: {max_compat}')
 
 
-class PatchedJSONSchema(marshmallow_jsonschema.JSONSchema):
-
-    # Patch marshmallow-jsonschema to support marshmallow-dataclass[union]
-    def _get_schema_for_field(self, obj, field):
-        """Patch marshmallow_jsonschema.base.JSONSchema to support marshmallow-dataclass[union]."""
-        if isinstance(field, marshmallow_fields.Raw) and field.allow_none and not field.validate:
-            # raw fields shouldn't be type string but type any. bug in marshmallow_dataclass:__init__.py:
-            #  if typ is Any:
-            #      metadata.setdefault("allow_none", True)
-            #      return marshmallow.fields.Raw(**metadata)
-            return {"type": ["string", "number", "object", "array", "boolean", "null"]}
-
-        if isinstance(field, marshmallow_dataclass.union_field.Union):
-            # convert to marshmallow_union.Union
-            field = marshmallow_union.Union([subfield for _, subfield in field.union_fields],
-                                            metadata=field.metadata,
-                                            required=field.required, name=field.name,
-                                            parent=field.parent, root=field.root, error_messages=field.error_messages,
-                                            default_error_messages=field.default_error_messages, default=field.default,
-                                            allow_none=field.allow_none)
-
-        return super()._get_schema_for_field(obj, field)
+# class PatchedJSONSchema(marshmallow_jsonschema.JSONSchema):
+
+#     # Patch marshmallow-jsonschema to support marshmallow-dataclass[union]
+#     def _get_schema_for_field(self, obj, field):
+#         """Patch marshmallow_jsonschema.base.JSONSchema to support marshmallow-dataclass[union]."""
+#         if isinstance(field, marshmallow_fields.Raw) and field.allow_none and not field.validate:
+#             # raw fields shouldn't be type string but type any. bug in marshmallow_dataclass:__init__.py:
+#             #  if typ is Any:
+#             #      metadata.setdefault("allow_none", True)
+#             #      return marshmallow.fields.Raw(**metadata)
+#             return {"type": ["string", "number", "object", "array", "boolean", "null"]}
+
+#         if isinstance(field, marshmallow_dataclass.union_field.Union):
+#             # convert to marshmallow_union.Union
+#             field = marshmallow_union.Union([subfield for _, subfield in field.union_fields],
+#                                             metadata=field.metadata,
+#                                             required=field.required, name=field.name,
+#                                             parent=field.parent, root=field.root, error_messages=field.error_messages,
+#                                             default_error_messages=field.default_error_messages, default=field.default,
+#                                             allow_none=field.allow_none)
+
+#         return super()._get_schema_for_field(obj, field)

With a little more investigation and testing (e.g. running the make test-cli and make test-cli-remote commands) we may be able to just remove this dependency altogether.

Justin Ibarra · Answer 5 · Thu Oct 10 2024 01:29:08 GMT+0800 (China Standard Time)

It's only use is when building API schemas, when updating stack versions to preserve the state in JSL