Giters

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

Home Page:https://clear.ml/docs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Adding clearml causes segmentation fault in unit tests with mypy

amishurov opened this issue · comments

commented

Describe the bug

Hello,
We have mypy typing tests included into unit tests
After importing clearml into work project, get constantly segfault error when run mypy tests

To reproduce

  1. clone repo https://github.com/amishurov/clearml_seg_fault (prepared it to easy peproduce bug)
  2. poetry install
  3. poetry run pytest
configfile: pyproject.toml
testpaths: tests
collected 2 items                                                                                                                                                          

tests/test_run.py .                                                                                                                                                  [ 50%]
tests/test_typing.py Fatal Python error: Segmentation fault

Current thread 0x00007f6947c70000 (most recent call first):
  File "/home/doc/.cache/pypoetry/virtualenvs/clearml-seg-fault-imUy90xh-py3.9/lib/python3.9/site-packages/mypy/split_namespace.py", line 31 in __getattr__
......
  File "/home/doc/.cache/pypoetry/virtualenvs/clearml-seg-fault-imUy90xh-py3.9/lib/python3.9/site-packages/mypy/split_namespace.py", line 31 in __getattr__
  File "/home/doc/.cache/pypoetry/virtualenvs/clearml-seg-fault-imUy90xh-py3.9/lib/python3.9/site-packages/mypy/split_namespace.py", line 31 in __getattr__
  ...
Segmentation fault (core dumped)

Expected behaviour

Unit tests with mypy should work without errors

Environment

  • Server type self hosted
  • ClearML SDK Version 1.14.4
  • ClearML Server Version 1.14.4
  • Python Version 3.9
  • OS Linux

Related Discussion

I suggest problem is clearml patches argparse module and mypy/split_namespace.py calls recursive in patched argparse

commented

Hi @amishurov,
This is indeed odd, notice that if you mark out the clearml import in run.py you will get an error because the pyproject.toml has a typo, it should be:

[tool.mypy]
files = ["main/run.py", "tests"]

Back to the segmentation fault, it looks like a mypy issue? we will be happy to help though...

commented

thank you, fixed pyproject.toml
Created issue in clearml project because

commented

clearml patches argparse https://github.com/allegroai/clearml/blob/master/clearml/binding/args.py#L251

Why would you attribute it to patching argparser?
Patching a python function can cause an exception not segmentation fault...
Could you open the issue on mypy and link to this one so that we can work with them?

commented

Why would you attribute it to patching argparser?

Added branch with "fix" to my demo example above:
https://github.com/amishurov/clearml_seg_fault/compare/segfault_fixed?expand=1
With reimport original argparse before executing mypy tests, there is no segafault happens

Could you open the issue on mypy and link to this one so that we can work with them?

Ok, come back with link

commented

Added branch with "fix" to my demo example above:
https://github.com/amishurov/clearml_seg_fault/compare/segfault_fixed?expand=1
With reimport original argparse before executing mypy tests, there is no segafault happens

So are you saying we should detect mypy running the code and disable the argparse patching at import time?

commented

As maintainer you feel free just to close this issue. And you shouldn't detect mypy running code obviously.