ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models

Home Page:http://ludwig.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unpin DeepSpeed to allow 0.13.0 and greater

arnavgarg1 opened this issue · comments

DeepSpeed 0.13.0 was released on Jan 20th, 2024, and is to be causing our tests to fail with the following error: https://github.com/ludwig-ai/ludwig/actions/runs/7614778487/job/20737857305?pr=3901

Collecting deepspeed!=0.11.0 (from ludwig==0.9.dev0)
  Downloading deepspeed-0.13.0.tar.gz (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 92.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [38 lines of output]
Warning: RNING] Unable to import torch, pre-compiling ops will be disabled. Please visit https://pytorch.org/ to see how to properly install torch on your system.
       [WARNING]  unable to import torch, please install it if you want to pre-compile any deepspeed ops.
      Traceback (most recent call last):
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/xpu/builder.py", line 14, in <module>
          from op_builder.builder import OpBuilder, TORCH_MAJOR, TORCH_MINOR
      ImportError: cannot import name 'TORCH_MAJOR' from 'op_builder.builder' (/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/builder.py)
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-l1qaau3a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-l1qaau3a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-l1qaau3a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-l1qaau3a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 37, in <module>
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/__init__.py", line 48, in <module>
          module = importlib.import_module(f".{module_name}", package=op_builder_dir)
        File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/xpu/__init__.py", line 6, in <module>
          from .cpu_adam import CPUAdamBuilder
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/xpu/cpu_adam.py", line 6, in <module>
          from .builder import SYCLOpBuilder
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/op_builder/xpu/builder.py", line 16, in <module>
          from deepspeed.ops.op_builder.builder import OpBuilder, TORCH_MAJOR, TORCH_MINOR
        File "/tmp/pip-install-e8btlrcw/deepspeed_59dc3579cb9240e9b578a8bf59239b19/deepspeed/__init__.py", line 10, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

Deepspeed was pinned to <0.13.0 in #3906