I try to pytest but it complains that "AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'"

Question

I try to pytest but it complains that "AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'"

otakutyrant opened this issue 4 months ago · comments

In the project, I executed PYTHONPATH=python pytest python/tests/. The result was:

===================================================================================== test session starts ==================================================================================
====
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/otakutyrant/abs/ctranslate2/src/CTranslate2/python
plugins: anyio-4.3.0, typeguard-4.2.1
collected 0 items / 9 errors                                                                                                                                                                
   

============================================================================================ ERRORS ========================================================================================
====
____________________________________________________________________________ ERROR collecting tests/test_fairseq.py ________________________________________________________________________
____
python/tests/test_fairseq.py:4: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
____________________________________________________________________________ ERROR collecting tests/test_marian.py _________________________________________________________________________
____
python/tests/test_marian.py:3: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
__________________________________________________________________________ ERROR collecting tests/test_opennmt_py.py _______________________________________________________________________
____
python/tests/test_opennmt_py.py:4: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
__________________________________________________________________________ ERROR collecting tests/test_opennmt_tf.py _______________________________________________________________________
____
ImportError while importing test module '/home/otakutyrant/abs/ctranslate2/src/CTranslate2/python/tests/test_opennmt_tf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
python/tests/test_opennmt_tf.py:4: in <module>
    import opennmt
E   ModuleNotFoundError: No module named 'opennmt'
_____________________________________________________________________________ ERROR collecting tests/test_spec.py __________________________________________________________________________
____
python/tests/test_spec.py:3: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
_________________________________________________________________________ ERROR collecting tests/test_storage_view.py ______________________________________________________________________
____
python/tests/test_storage_view.py:5: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
_________________________________________________________________________ ERROR collecting tests/test_transformers.py ______________________________________________________________________
____
python/tests/test_transformers.py:8: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
__________________________________________________________________________ ERROR collecting tests/test_translator.py _______________________________________________________________________
____
python/tests/test_translator.py:10: in <module>
    import test_utils
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
_____________________________________________________________________________ ERROR collecting tests/test_utils.py _________________________________________________________________________
____
python/tests/test_utils.py:43: in <module>
    ctranslate2.get_cuda_device_count() == 0, reason="Test case requires a CUDA device"
E   AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
=================================================================================== short test summary info ================================================================================
====
ERROR python/tests/test_fairseq.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_marian.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_opennmt_py.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_opennmt_tf.py
ERROR python/tests/test_spec.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_storage_view.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_transformers.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_translator.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
ERROR python/tests/test_utils.py - AttributeError: module 'ctranslate2' has no attribute 'get_cuda_device_count'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
====================================================================================== 9 errors in 1.48s ===================================================================================
====

Maybe I missed something. Could you tell me why please?

Minh-Thuc · Answer 1 · Wed Apr 03 2024 19:57:10 GMT+0800 (China Standard Time)

Can you describe more in detail how you installed ctranslate2?
Try to install ctranslate2 with pip and in the repo Ctranslate2, run this:

pytest python/tests

If you install ctranslate2 in a virtual environment, make sure the used pytest is installed in the virtual environment.

otakutyrant · Answer 2 · Wed Apr 03 2024 23:32:07 GMT+0800 (China Standard Time)

As for your question. I installed ctranslate2 via this PKGBUILD, which is used by Arch Linux to download source code, build it, package it and install it finally. Arch Linux does not allow users execute pip install xxx to install packages system-wide unless users execute the pip command in a virtual environment.

I have installed ctranslate2 in my Arch Linux already, and the result ot pytest python-tests is:

    ~/P/CTranslate2   master  ✚1* pytest python/tests              23617  11:22:36
 PM 
===================================== test session starts =================================
====
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/otakutyrant/Projects/CTranslate2/python
plugins: anyio-4.3.0, typeguard-4.2.1
collected 185 items / 1 error                                                              
   

=========================================== ERRORS ========================================
====
__________________________ ERROR collecting tests/test_opennmt_tf.py ______________________
____
ImportError while importing test module '/home/otakutyrant/Projects/CTranslate2/python/test
s/te
st_opennmt_tf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
python/tests/test_opennmt_tf.py:4: in <module>
    import opennmt
E   ModuleNotFoundError: No module named 'opennmt'
=================================== short test summary info ===============================
====
ERROR python/tests/test_opennmt_tf.py
!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!
!!!!
====================================== 1 error in 1.23s ===================================
====

Now I created a virtual environment and have pip installed ctranslate2 and pytest, the result of pytest python/tests is:

23:29:24  otakutyrant@ArchPC  ~/Projects/CTranslate2  🐍 .env   master ✘ ✹ ✭ 
$ pytest python/tests
===================================== test session starts =================================
====
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/otakutyrant/Projects/CTranslate2/python
plugins: anyio-4.3.0, typeguard-4.2.1
collected 185 items / 1 error                                                              
   

=========================================== ERRORS ========================================
====
__________________________ ERROR collecting tests/test_opennmt_tf.py ______________________
____
ImportError while importing test module '/home/otakutyrant/Projects/CTranslate2/python/test
s/te
st_opennmt_tf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
python/tests/test_opennmt_tf.py:4: in <module>
    import opennmt
E   ModuleNotFoundError: No module named 'opennmt'
=================================== short test summary info ===============================
====
ERROR python/tests/test_opennmt_tf.py
!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!
!!!!
====================================== 1 error in 1.25s ===================================
====

Minh-Thuc · Answer 3 · Wed Apr 03 2024 23:53:34 GMT+0800 (China Standard Time)

As showing in the message error, you haven't installed the OpenNMT-py. It is normal that in the PKGBUILD, I don't see OpenNMT-py because this is used only for the tests. I think you have to add it in the PKGBUILD (I don't know much how could we install python packages in Arch Linux).

otakutyrant · Answer 4 · Thu Apr 04 2024 00:07:20 GMT+0800 (China Standard Time)

Oh, so opennmt is a dependency I missed. It is midnight here. I try to install opennmt by myself in 24 hours.

otakutyrant · Answer 5 · Thu Apr 04 2024 15:17:49 GMT+0800 (China Standard Time)

I find out that it is hard to test, and I think the dependencies of opennmt-py and opennmt-tf are conflict even. Sorry, I give up to test, and actually I have solved a real X problem.