Error while running the tutorial

[Artur Szałata]

Executing cell 5 in tutorial results in core dump. Tried to execute the same lines in a script and got Illegal instruction (core dumped).

Any help is appreciated!

The machine has 90GB of RAM.
Python 3.7.16

pip freeze

annoy==1.17.1
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1666191106763/work/dist
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1649500320262/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1671632566681/work
Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1667688356751/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1649463573192/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1674535352125/work
brotlipy @ file:///home/conda/feedstock_root/build_artifacts/brotlipy_1648854164153/work
certifi==2022.12.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1636046052501/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1661170624537/work
colorama==0.4.6
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1637687023717/work
cycler==0.11.0
debugpy==1.6.6
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
dnspython==2.3.0
docopt==0.6.2
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1663619548554/work/dist
flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1667734568827/work/source/flit_core
fonttools==4.38.0
gitdb==4.0.10
GitPython==3.1.30
h5py==3.8.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1653252814274/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1672681417544/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1620912939357/work/dist/ipykernel-5.5.5-py3-none-any.whl
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1651240553635/work
ipython-genutils==0.2.0
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
joblib==1.2.0
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work
jsonpickle==1.5.2
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1669810440410/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1633454794268/work
jupyter-server @ file:///croot/jupyter_server_1671707632269/work
jupyter_core @ file:///opt/conda/conda-bld/jupyter_core_1664917302524/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1674494302491/work
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1649936611996/work
jupyterlab_server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1673884094295/work
kiwisolver==1.4.4
llvmlite==0.35.0
MarkupSafe @ file:///opt/conda/conda-bld/markupsafe_1654597864307/work
matplotlib==3.5.3
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1657892024508/work
MultiMAP @ file:///home/icb/artur.szalata/code/MultiMAP
munch==2.5.0
natsort==8.2.0
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1667492839781/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1662750566673/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1673893169158/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1673560067442/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx==2.6.3
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1667565639349/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1667478401171/work
numba==0.52.0
numpy==1.21.6
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1673482170163/work
pandas==1.3.5
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
patsy==0.5.3
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow==9.4.0
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1633981968097/work
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1674535637125/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1670414775770/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py-cpuinfo==9.0.0
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1672682006896/work
pymongo==4.3.3
pynndescent==0.5.8
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==3.0.9
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1636098896055/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1648857264451/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1673864280276/work
PyYAML==6.0
pyzmq==19.0.2
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1673863902341/work
sacred==0.8.2
scanpy==1.9.1
scikit-learn==1.0.2
scipy==1.7.3
seaborn==0.12.2
seml==0.3.7
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work
session-info==1.0.0
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smmap==5.0.0
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1658207591808/work
statsmodels==0.13.5
stdlib-list==0.8.0
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1670253674810/work
threadpoolctl==3.1.0
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1648827244717/work
tqdm==4.64.1
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1673359992537/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1665144421445/work
umap-learn==0.5.3
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1673452138552/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
webencodings==0.5.1
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1667568040382/work
wrapt==1.14.1
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1669453021653/work``` 

[Krzysztof Polanski]

I saw you respond in a prior issue which ended up touching on M1. Are you on an M1 mac by chance?

[Artur Szałata]

Thanks for the quick response!
I tried first on M1 Mac but as I noted there I realized that there are dependency issues on Python >=3.8.

Here I report issues on Linux kernel #1 SMP Wed Nov 9 20:13:27 UTC 2022. Installation works, but the execution of MultiMAP.Integration([rna, atac_genes], ['X_pca', 'X_lsi']) consistently fails with core dump.

[Krzysztof Polanski]

Could I get you to try a thing for me? I'm pretty sure the various weird crashes people are encountering ultimately lead back to annoy, a package that lives up to its name. Could you pop open MultiMAP/matrix.py in your clone of the repo and replace the definitions of create_tree() and query_tree() with the following:

def create_tree(data, metric):
    if metric == 'euclidean':
        ckd = cKDTree(data)
    else:
        ckd = KDTree(data, metric=metric)
    return ckd

def query_tree(data, ckd, k, metric):
    if metric == 'euclidean':
        ckdout = ckd.query(x=data, k=k, n_jobs=-1)
    else:
        ckdout = ckd.query(data, k=k)
    return ckdout

Then go back to the main directory of the clone and call pip3 install . again, and try to run the thing again.

[Artur Szałata]

indeed, that fixes the issue, thank you! :)

[Krzysztof Polanski]

Great, in that case I'll roll this out properly (need to decide whether to allow annoy at all). I also figured out how to get rid of the numba requirement, allowing the package to move into newer python releases - as it turns out, it was a single problematic function borrowed from an archival UMAP release.