Kyber does not appear to work on Python 3.11.6, MacOS 14.3 on M1 Mac
jathanism opened this issue · comments
Using quantcrypt 0.3.3.
Was following the docs to play around and ran into this issue:
In [21]: from quantcrypt import (
...: kem, # Key Encapsulation Mechanism algos - public-key cryptography
...: dss, # Digital Signature Scheme algos - secret-key signatures
...: cipher, # The Krypton Cipher - symmetric cipher based on AES-256
...: kdf, # Argon2 helpers + KMAC-KDF - key derivation functions
...: errors, # All errors QuantCrypt may raise - also available from other modules
...: utils # Helper utilities from all modules - gathered into one module
...: )
In [22]: k = kem.Kyber()
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/quantcrypt/internal/pqa/common.py:75, in BasePQAlgorithm.__init__(self, variant)
74 _var = variant or PQAVariant.AVX2
---> 75 self._lib = self._import(_var)
76 self.variant = _var
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/quantcrypt/internal/pqa/common.py:66, in BasePQAlgorithm._import(self, variant)
64 @lru_cache
65 def _import(self, variant: PQAVariant) -> ModuleType:
---> 66 return importlib.import_module(
67 f"quantcrypt.internal.bin.{platform.system()}" +
68 f".{variant.value}.{self.name.replace('-', '_')}"
69 ).lib
File ~/.pyenv/versions/3.11.6/lib/python3.11/importlib/__init__.py:126, in import_module(name, package)
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)
File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1126, in _find_and_load_unlocked(name, import_)
File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)
File <frozen importlib._bootstrap>:1204, in _gcd_import(name, package, level)
File <frozen importlib._bootstrap>:1176, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1140, in _find_and_load_unlocked(name, import_)
ModuleNotFoundError: No module named 'quantcrypt.internal.bin.Darwin.avx2'
During handling of the above exception, another exception occurred:
SystemExit Traceback (most recent call last)
[... skipping hidden 1 frame]
Cell In[22], line 1
----> 1 k = kem.Kyber()
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/pydantic/validate_call_decorator.py:58, in validate_call.<locals>.validate.<locals>.wrapper_function(*args, **kwargs)
56 @functools.wraps(function)
57 def wrapper_function(*args, **kwargs):
---> 58 return validate_call_wrapper(*args, **kwargs)
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py:81, in ValidateCallWrapper.__call__(self, *args, **kwargs)
80 def __call__(self, *args: Any, **kwargs: Any) -> Any:
---> 81 res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
82 if self.__return_pydantic_validator__:
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/quantcrypt/internal/pqa/kem.py:144, in Kyber.__init__(self, variant)
127 """
128 Initializes the Kyber instance with C extension binaries.
129 User is able to override which underlying binary is used for the
(...)
142 should have CLEAN binaries available.
143 """
--> 144 super().__init__(variant)
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/quantcrypt/internal/pqa/common.py:87, in BasePQAlgorithm.__init__(self, variant)
86 raise ex
---> 87 raise SystemExit( # pragma: no cover
88 "Quantcrypt Fatal Error:\n"
89 "Unable to continue due to missing CLEAN binaries."
90 )
SystemExit: Quantcrypt Fatal Error:
Unable to continue due to missing CLEAN binaries.
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
[... skipping hidden 1 frame]
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2121, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
2118 if exception_only:
2119 stb = ['An exception has occurred, use %tb to see '
2120 'the full traceback.\n']
-> 2121 stb.extend(self.InteractiveTB.get_exception_only(etype,
2122 value))
2123 else:
2125 def contains_exceptiongroup(val):
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:710, in ListTB.get_exception_only(self, etype, value)
702 def get_exception_only(self, etype, value):
703 """Only print the exception type and message, without a traceback.
704
705 Parameters
(...)
708 value : exception value
709 """
--> 710 return ListTB.structured_traceback(self, etype, value)
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:568, in ListTB.structured_traceback(self, etype, evalue, etb, tb_offset, context)
565 chained_exc_ids.add(id(exception[1]))
566 chained_exceptions_tb_offset = 0
567 out_list = (
--> 568 self.structured_traceback(
569 etype,
570 evalue,
571 (etb, chained_exc_ids), # type: ignore
572 chained_exceptions_tb_offset,
573 context,
574 )
575 + chained_exception_message
576 + out_list)
578 return out_list
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:1435, in AutoFormattedTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
1433 else:
1434 self.tb = etb
-> 1435 return FormattedTB.structured_traceback(
1436 self, etype, evalue, etb, tb_offset, number_of_lines_of_context
1437 )
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:1326, in FormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1323 mode = self.mode
1324 if mode in self.verbose_modes:
1325 # Verbose modes need a full traceback
-> 1326 return VerboseTB.structured_traceback(
1327 self, etype, value, tb, tb_offset, number_of_lines_of_context
1328 )
1329 elif mode == 'Minimal':
1330 return ListTB.get_exception_only(self, etype, value)
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:1173, in VerboseTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
1164 def structured_traceback(
1165 self,
1166 etype: type,
(...)
1170 number_of_lines_of_context: int = 5,
1171 ):
1172 """Return a nice text document describing the traceback."""
-> 1173 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
1174 tb_offset)
1176 colors = self.Colors # just a shorthand + quicker name lookup
1177 colorsnormal = colors.Normal # used a lot
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:1063, in VerboseTB.format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
1060 assert isinstance(tb_offset, int)
1061 head = self.prepare_header(str(etype), self.long_header)
1062 records = (
-> 1063 self.get_records(etb, number_of_lines_of_context, tb_offset) if etb else []
1064 )
1066 frames = []
1067 skipped = 0
File ~/Library/Caches/pypoetry/virtualenvs/bi-sdk-eiGFesGp-py3.11/lib/python3.11/site-packages/IPython/core/ultratb.py:1131, in VerboseTB.get_records(self, etb, number_of_lines_of_context, tb_offset)
1129 while cf is not None:
1130 try:
-> 1131 mod = inspect.getmodule(cf.tb_frame)
1132 if mod is not None:
1133 mod_name = mod.__name__
AttributeError: 'tuple' object has no attribute 'tb_frame'
In [23]:
@jathanism
Hey, thank you for trying out QuantCrypt! Your avatar image sure scared me at first! :P
The issue you're encountering stems from the fact that prior to library version 0.3.4, the binaries built by GitHub actions used Python 3.10, which meant that the compiled binaries were only compatible with Python 3.10 (see closed issue #1).
At that time, I was not aware that CFFI compiled version-specific binaries. The latest version of QuantCrypt compiles the binaries with Python 3.12 and the library is marked to support only Python 3.12. This constriction ensures that when you install the library, the binaries are guaranteed to work with the version of Python that you are using. It would be possible to support multiple Python versions simultaneously, but this would require multiplying the package size by the size of the additional compiled binaries for all platforms and I'm not certain it would be worthwhile to do.
If you have feedback or advice on the course of actions to take on this matter, I would be grateful for your input! In the meantime, I would kindly ask you to use the latest version of QuantCrypt with Python version 3.12.
EDIT: I simplified the solution to this issue by yanking all releases prior to version 0.3.4, so now it should be impossible to unintentionally install an incompatible version of this library.
Whoops sorry about the delayed response. I supposed the only real solution for supporting multiple Python versions would be to build them in CI and then upload a .whl for each one. Maintaining libraries is hard. Thanks for the response! :)