sparse.tensordot in v0.11 and 0.12 introduces Numba typing errors
sametz opened this issue · comments
In Issue #493 I showed a case where sparse.tensordot fails a NumPy documentation example when both array arguments are type np.ndarray.
Here, I have an example of a tensordot function that worked for sparse v0.10.0 and earlier but fails for v0.11 and 0.12. If one or both of the arrays are type COO, Numba typing errors result.
The following test is parametrized for different combinations of ndarray vs. COO argument types:
import numpy as np
import pytest
import sparse
v2 =np.array([10.0, 20.0])
Lz2 = np.array(
[[[0.5 + 0.j, 0. + 0.j, 0. + 0.j, 0. + 0.j],
[0. + 0.j, 0.5 + 0.j, 0. + 0.j, 0. + 0.j],
[0. + 0.j, 0. + 0.j, -0.5 + 0.j, -0. + 0.j],
[0. + 0.j, 0. + 0.j, -0. + 0.j, -0.5 + 0.j]],
[[0.5 + 0.j, 0. + 0.j, 0. + 0.j, 0. + 0.j],
[0. + 0.j, -0.5 + 0.j, 0. + 0.j, -0. + 0.j],
[0. + 0.j, 0. + 0.j, 0.5 + 0.j, 0. + 0.j],
[0. + 0.j, -0. + 0.j, 0. + 0.j, -0.5 + 0.j]]]
)
v2_coo = sparse.COO(v2)
Lz2_coo = sparse.COO(Lz2)
@pytest.mark.parametrize("v, Lz", [
(v2, Lz2),
(v2_coo, Lz2),
(v2, Lz2_coo),
(v2_coo, Lz2_coo)
])
def test_twospin_v_Lz(v, Lz):
H = sparse.tensordot(v, Lz, axes=1)
if not isinstance(H, np.ndarray):
H = H.todense()
assert np.allclose(H,
np.array(
[[15. + 0.j, 0. + 0.j, 0. + 0.j, 0. + 0.j],
[0. + 0.j, -5. + 0.j, 0. + 0.j, 0. + 0.j],
[0. + 0.j, 0. + 0.j, 5. + 0.j, 0. + 0.j],
[0. + 0.j, 0. + 0.j, 0. + 0.j, -15. + 0.j]]
))The case where both arrays are type ndarray fails as previously described:
with an error coming from the sparse.tensordot function:
at = a.transpose(newaxes_a).reshape(newshape_a)
bt = b.transpose(newaxes_b).reshape(newshape_b)
res = _dot(at, bt)
> return res.reshape(olda + oldb)
E AttributeError: 'NoneType' object has no attribute 'reshape'
../../../.conda/envs/nmrsim/lib/python3.7/site-packages/sparse/_coo/common.py:172: AttributeError
The other three tests pass up to sparse 0.10.0, but fail in 0.11.0 and 0.12.0.
When one of the tensordot arguments is type np.ndarray, and the other COO,
the following numba typing error results:
v = <COO: shape=(2,), dtype=float64, nnz=2, fill_value=0.0>
Lz = array([[[ 0.5+0.j, 0. +0.j, 0. +0.j, 0. +0.j],
[ 0. +0.j, 0.5+0.j, 0. +0.j, 0. +0.j],
[ 0. +0.j,... +0.j, 0. +0.j],
[ 0. +0.j, 0. +0.j, 0.5+0.j, 0. +0.j],
[ 0. +0.j, 0. +0.j, 0. +0.j, -0.5+0.j]]])
@pytest.mark.parametrize("v, Lz", [
(v2, Lz2),
(v2_coo, Lz2),
(v2, Lz2_coo),
(v2_coo, Lz2_coo)
])
def test_twospin_v_Lz(v, Lz):
> H = sparse.tensordot(v, Lz, axes=1)
tests/test_sparse.py:130:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py:145: in tensordot
res = _dot(at, bt, return_type)
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py:423: in _dot
return _dot_coo_ndarray_type(a.dtype, b.dtype)(
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/numba/core/dispatcher.py:420: in _compile_for_args
error_rewrite(e, 'typing')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
e = TypingError('Failed in nopython mode pipeline (step: nopython frontend)\nNo implementation of function Function(<built...1:\n out[oidx1, oidx2] += data1[didx1] * array2[oidx2, coords1[1, didx1]]\n ^\n')
issue_type = 'typing'
def error_rewrite(e, issue_type):
"""
Rewrite and raise Exception `e` with help supplied based on the
specified issue_type.
"""
if config.SHOW_HELP:
help_msg = errors.error_extras[issue_type]
e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
if config.FULL_TRACEBACKS:
raise e
else:
> raise e.with_traceback(None)
E numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
E No implementation of function Function(<built-in function setitem>) found for signature:
E
E >>> setitem(array(float64, 2d, C), UniTuple(int64 x 2), complex128)
E
E There are 16 candidate implementations:
E - Of which 16 did not match due to:
E Overload of function 'setitem': File: <numerous>: Line N/A.
E With argument(s): '(array(float64, 2d, C), UniTuple(int64 x 2), complex128)':
E No match.
E
E During: typing of setitem at /Users/geoffreysametz/anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py (1026)
E
E File "../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py", line 1026:
E def _dot_coo_ndarray(coords1, data1, array2, out_shape): # pragma: no cover
E <source elided>
E while didx1 < len(data1) and coords1[0, didx1] == oidx1:
E out[oidx1, oidx2] += data1[didx1] * array2[oidx2, coords1[1, didx1]]
E ^
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/numba/core/dispatcher.py:361: TypingError
When both arrays are type COO, a different numba typing error results:
v = <COO: shape=(2,), dtype=float64, nnz=2, fill_value=0.0>, Lz = <COO: shape=(2, 4, 4), dtype=complex128, nnz=8, fill_value=0j>
@pytest.mark.parametrize("v, Lz", [
(v2, Lz2),
(v2_coo, Lz2),
(v2, Lz2_coo),
(v2_coo, Lz2_coo)
])
def test_twospin_v_Lz(v, Lz):
> H = sparse.tensordot(v, Lz, axes=1)
tests/test_sparse.py:130:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py:145: in tensordot
res = _dot(at, bt, return_type)
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py:401: in _dot
coords, data = _dot_coo_coo_type(a.dtype, b.dtype)(
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/numba/core/dispatcher.py:420: in _compile_for_args
error_rewrite(e, 'typing')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
e = TypingError('Failed in nopython mode pipeline (step: nopython frontend)\nNo implementation of function Function(<built...oo_coo(\n <source elided>\n ):\n sums[k] += av * bv\n ^\n')
issue_type = 'typing'
def error_rewrite(e, issue_type):
"""
Rewrite and raise Exception `e` with help supplied based on the
specified issue_type.
"""
if config.SHOW_HELP:
help_msg = errors.error_extras[issue_type]
e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
if config.FULL_TRACEBACKS:
raise e
else:
> raise e.with_traceback(None)
E numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
E No implementation of function Function(<built-in function setitem>) found for signature:
E
E >>> setitem(array(float64, 1d, C), int64, complex128)
E
E There are 16 candidate implementations:
E - Of which 16 did not match due to:
E Overload of function 'setitem': File: <numerous>: Line N/A.
E With argument(s): '(array(float64, 1d, C), int64, complex128)':
E No match.
E
E During: typing of setitem at /Users/geoffreysametz/anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py (969)
E
E File "../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/sparse/_common.py", line 969:
E def _dot_coo_coo(
E <source elided>
E ):
E sums[k] += av * bv
E ^
../../../anaconda3/anaconda3/envs/nmrsim_39/lib/python3.9/site-packages/numba/core/dispatcher.py:361: TypingError
Expected Behavior
- expected the changes to
sparse.tensordotin sparse v0.11 to be backwards-compatible and not break pre-existing, working code.
System
This has been tested on Mac Catalina and Windows 10 (see examples in Issue #493). Here are two combinations re-verified on the Mac where tests pass with Sparse 0.10 but fail for Sparse 0.12.
Failing system config:
- OS and version: Mac Catalina 10.15.7
sparseversion 0.12.0- NumPy version 1.20.3
- Numba version 0.53.1
Passing 3/4 tests (i.e. excluding Issue #493 case) system config:
- OS and version: Mac Catalina 10.15.7
sparseversion 0.10.0- NumPy version 1.18.1
- Numba version 0.49.1
Additional context
This failure is also noticed in linux cloud servers, e.g. when a Binder launches to run a Jupyter notebook hosted on GitHub. So, it is OS-independent.
@daletovar worked on the dot code last, I believe.