AttributeError in `03-matrix-multiplication.ipynb` tutorial
syadegari opened this issue · comments
Description
Encountered an AttributeError in the 03-matrix-multiplication.ipynb tutorial when attempting to auto-tune configurations for matrix multiplication using Triton.
Error Message
AttributeError: CudaDriver object has no attribute 'active'
Steps to Reproduce
- Run the first code cell of the notebook.
- The error is triggered when attempting to access
triton.runtime.driver.active.get_current_target().
Temporary Fix
I can bypass (not a good fix, but at least runs the notebook and outputs are the same as then ones shown in the online version of tutorial) the error by modifying the device detection and configuration retrieval functions as follows:
def is_cuda():
return torch.cuda.is_available()
def is_hip_mi200():
return not is_cuda()
def get_autotune_config():
if is_cuda():
return get_cuda_autotune_config()
else:
return get_hip_autotune_config()I appreciate help/tips for a better fix.
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly
and things should work as is, no need to change the code.
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightlyand things should work as is, no need to change the code.
I can't install it correct, pip is looking for the correct package, the logger is as follows:
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/
Collecting triton-nightly
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240605045615/triton_nightly-3.0.0.post20240605045615-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.2/136.2 MB 2.3 MB/s eta 0:00:00
Requirement already satisfied: filelock in ./anaconda3/envs/triton-nightly/lib/python3.9/site-packages (from triton-nightly) (3.14.0)
INFO: pip is looking at multiple versions of triton-nightly to determine which version is compatible with other requirements. This could take a while.
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240604012636/triton_nightly-3.0.0.post20240604012636-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240603025518/triton_nightly-3.0.0.post20240603025518-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 2.8 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240601044641/triton_nightly-3.0.0.post20240601044641-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.0 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240531062509/triton_nightly-3.0.0.post20240531062509-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 2.9 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240530020152/triton_nightly-3.0.0.post20240530020152-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240529030827/triton_nightly-3.0.0.post20240529030827-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.1/136.1 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240527145827/triton_nightly-3.0.0.post20240527145827-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.7/135.7 MB 3.0 MB/s eta 0:00:00
INFO: pip is still looking at multiple versions of triton-nightly to determine which version is compatible with other requirements. This could take a while.
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240526215807/triton_nightly-3.0.0.post20240526215807-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 3.4 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240524232251/triton_nightly-3.0.0.post20240524232251-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 2.8 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240524063540/triton_nightly-3.0.0.post20240524063540-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240522224832/triton_nightly-3.0.0.post20240522224832-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 2.4 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240522054702/triton_nightly-3.0.0.post20240522054702-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.4/135.4 MB 3.3 MB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240521001803/triton_nightly-3.0.0.post20240521001803-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.4/135.4 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240520020804/triton_nightly-3.0.0.post20240520020804-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.8/135.8 MB 2.2 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240515174524/triton_nightly-3.0.0.post20240515174524-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.2/135.2 MB 3.2 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240513074433/triton_nightly-3.0.0.post20240513074433-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.8/136.8 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240512011919/triton_nightly-3.0.0.post20240512011919-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.8/136.8 MB 3.3 MB/s eta 0:00:00
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240511014100/triton_nightly-3.0.0.post20240511014100-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.1/136.8 MB 3.0 MB/s eta 0:00:36
I think this is due to the library llnl-hatchet not being available in pip. Probably related to: LLNL/hatchet#134
I did the following to make it work.
pip install git+https://github.com/LLNL/hatchet
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolver
I think the hatchet issue should have been fixed now
I think the hatchet issue should have been fixed now
Mmm, still does not work out of the box for python 3.11.
$ pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-
resolver
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ Collecting triton-nightly
Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240610003544/tri
ton_nightly-3.0.0.post20240610003544-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (138.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.5/138.5 MB 819.5 kB/s eta 0:00:00 Requirement already satisfied, skipping upgrade: filelock in ./miniconda3/envs/ml-nightly2/lib/python3.11/site-packages (from triton-nightly) (3.13.1)
ERROR: Could not find a version that satisfies the requirement llnl-hatchet (from triton-nightly) (from versions: none)
ERROR: No matching distribution found for llnl-hatchet (from triton-nightly)
$ python --version
Python 3.11.9 But if I do the following it works just fine:
pip install llnl-hatchet
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolverI think this is due to the library llnl-hatchet not being available in pip. Probably related to: LLNL/hatchet#134
I did the following to make it work.
pip install git+https://github.com/LLNL/hatchet pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolver
Thanks a lot, it works for me. My python environment is python==3.9.