Giters

horovod / horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

Home Page:http://horovod.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Tesnorflow2 examples won't run with more than 1 GPU

laytonjbgmail opened this issue · comments

commented

Environment:

  1. Framework: (TensorFlow, Keras, PyTorch, MXNet) TensorFlow
  2. Framework version: tensorflow-2.13.0
  3. Horovod version: horovod-0.28.1
  4. MPI version: openmpi-4.1.2
  5. CUDA version: 11.8
  6. NCCL version: nccl-2.182.12.1
  7. Python version: 3.8.12
  8. Spark / PySpark version: none
  9. Ray version: none
  10. OS and version: Ubuntu 22.04
  11. GCC version: 11.3.0
  12. CMake version: 3.25.0

Checklist:

  1. Did you search issues to find if somebody asked this question before? I did - I didn't find anything useful.
  2. If your question is about hang, did you read this doc? NA
  3. If your question is about docker, did you read this doc? NA
  4. Did you check if you question is answered in the troubleshooting guide? yes - not there

Bug report:
I tried running the ternsorflow2 example from github (https://github.com/horovod/horovod/blob/master/examples/tensorflow2/tensorflow2_keras_mnist.py) with 2 GPUs and I get a segfault (signal 11). Running with 1 GPU works correctly. The command line:

mpirun -np 2 -H laytonjb-APEXX-T3-04:1,laytonjb-APEXX-T3-04:2 -bind-to none --map-by slot -x NCCL_DEBUG=INFO python3 ./tensorflow2_keras_mnist.py

I tried removing all options except "-np 2" and "-H ..." and it results in the same error

The output from the command is below:

$ mpirun -np 2 -H laytonjb-APEXX-T3-04:1,laytonjb-APEXX-T3-04:2 -bind-to none --map-by slot -x NCCL_DEBUG=INFO python3 ./tensorflow2_keras_mnist.py
2023-08-01 13:21:14.848313: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-01 13:21:14.865791: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-01 13:21:15.299373: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-01 13:21:15.318233: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-01 13:21:15.868042: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.868481: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.871121: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.871471: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.885529: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.885820: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.886033: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.886238: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.899707: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.900097: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.900401: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:15.900698: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.118472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.118754: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.118960: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.148485: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.148920: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.149305: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.230149: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.230673: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.230872: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.231060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 47372 MB memory: -> device: 1, name: Quadro RTX 8000, pci bus id: 0000:49:00.0, compute capability: 7.5
2023-08-01 13:21:16.261160: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.261401: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.261599: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-01 13:21:16.261793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 46815 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:21:00.0, compute capability: 7.5
Epoch 1/24
2023-08-01 13:21:17.876383: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
2023-08-01 13:21:17.904957: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
2023-08-01 13:21:18.014518: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8902
2023-08-01 13:21:18.044255: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8902
[laytonjb-APEXX-T3-04:79057] *** Process received signal ***
[laytonjb-APEXX-T3-04:79057] Signal: Segmentation fault (11)
[laytonjb-APEXX-T3-04:79057] Signal code: Invalid permissions (2)
[laytonjb-APEXX-T3-04:79057] Failing at address: 0x7f411f371800
[laytonjb-APEXX-T3-04:79058] *** Process received signal ***
[laytonjb-APEXX-T3-04:79058] Signal: Segmentation fault (11)
[laytonjb-APEXX-T3-04:79058] Signal code: Invalid permissions (2)
[laytonjb-APEXX-T3-04:79058] Failing at address: 0x7f5a33372218
[laytonjb-APEXX-T3-04:79057] [ 0] [laytonjb-APEXX-T3-04:79058] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f451c842520]
[laytonjb-APEXX-T3-04:79057] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f5e35242520]
[laytonjb-APEXX-T3-04:79058] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x1a08c1)[0x7f451c9a08c1]
[laytonjb-APEXX-T3-04:79057] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x1a08c1)[0x7f5e353a08c1]
[laytonjb-APEXX-T3-04:79058] [ 2] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_btl_vader.so(+0x39e4)[0x7f5ddc00e9e4]
[laytonjb-APEXX-T3-04:79058] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_btl_vader.so(+0x39e4)[0x7f44bf6789e4]
[laytonjb-APEXX-T3-04:79057] [ 3] [ 3] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_prepare+0x45)[0x7f5dcfe43885]
[laytonjb-APEXX-T3-04:79058] [ 4] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x112d)[0x7f5dcfe3557d]
[laytonjb-APEXX-T3-04:79058] [ 5] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_prepare+0x45)[0x7f44bf66b885]
[laytonjb-APEXX-T3-04:79057] [ 4] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x112d)[0x7f44bf65d57d]
[laytonjb-APEXX-T3-04:79057] [ 5] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/libmpi.so.40(ompi_coll_base_allreduce_intra_ring+0x30b)[0x7f5dde571d9b]
[laytonjb-APEXX-T3-04:79058] [ 6] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allreduce_intra_dec_fixed+0x4f)[0x7f5dcf9a008f]
[laytonjb-APEXX-T3-04:79058] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/libmpi.so.40(ompi_coll_base_allreduce_intra_ring+0x30b)[0x7f44cf854d9b]
[laytonjb-APEXX-T3-04:79057] [ 6] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allreduce_intra_dec_fixed+0x4f)[0x7f44bf01908f]
[laytonjb-APEXX-T3-04:79057] [ 7] [ 7] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/libmpi.so.40(PMPI_Allreduce+0x131)[0x7f44cf80d421]
[laytonjb-APEXX-T3-04:79057] [ 8] /home/laytonjb/bin/GCC-11.3/OPENMPI-4.1.5/lib/libmpi.so.40(PMPI_Allreduce+0x131)[0x7f5dde52a421]
[laytonjb-APEXX-T3-04:79058] [ 8] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZN7horovod6common16MPI_GPUAllreduce7ExecuteERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseE+0x283)[0x7f44c5d22ce3]
[laytonjb-APEXX-T3-04:79057] [ 9] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZN7horovod6common16MPI_GPUAllreduce7ExecuteERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseE+0x283)[0x7f5dde722ce3]
[laytonjb-APEXX-T3-04:79058] [ 9] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZNK7horovod6common16OperationManager16ExecuteAllreduceERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseE+0x7d)[0x7f44c5ce5d1d]
[laytonjb-APEXX-T3-04:79057] [10] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZNK7horovod6common16OperationManager16ExecuteAllreduceERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseE+0x7d)[0x7f5dde6e5d1d]
[laytonjb-APEXX-T3-04:79058] [10] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZNK7horovod6common16OperationManager16ExecuteOperationERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseERNS0_10ProcessSetE+0x4c)[0x7f44c5ce61fc]
[laytonjb-APEXX-T3-04:79057] [11] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(_ZNK7horovod6common16OperationManager16ExecuteOperationERSt6vectorINS0_16TensorTableEntryESaIS3_EERKNS0_8ResponseERNS0_10ProcessSetE+0x4c)[0x7f5dde6e61fc]
[laytonjb-APEXX-T3-04:79058] [11] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(+0xb4c23)[0x7f44c5cb4c23]
[laytonjb-APEXX-T3-04:79057] [12] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/horovod/tensorflow/mpi_lib.cpython-39-x86_64-linux-gnu.so(+0xb4c23)[0x7f5dde6b4c23]
[laytonjb-APEXX-T3-04:79058] [12] /home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2(+0x1ac3e20)[0x7f5e330c3e20]
[laytonjb-APEXX-T3-04:79058] [13] /lib/x86_64-linux-gnu/libc.so.6(+0x94b43)[0x7f5e35294b43]
[laytonjb-APEXX-T3-04:79058] [14] /lib/x86_64-linux-gnu/libc.so.6(+0x126a00)[0x7f5e35326a00]
[laytonjb-APEXX-T3-04:79058] *** End of error message ***
/home/laytonjb/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2(+0x1ac3e20)[0x7f451a8c3e20]
[laytonjb-APEXX-T3-04:79057] [13] /lib/x86_64-linux-gnu/libc.so.6(+0x94b43)[0x7f451c894b43]
[laytonjb-APEXX-T3-04:79057] [14] /lib/x86_64-linux-gnu/libc.so.6(+0x126a00)[0x7f451c926a00]
[laytonjb-APEXX-T3-04:79057] *** End of error message ***

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun noticed that process rank 0 with PID 0 on node laytonjb-APEXX-T3-04 exited on signal 11 (Segmentation fault).

In fact, when I run any example in the tensorflow2 examples it segfaults when running more than 1 GPU.

I hate to make this report long, nut here is the output from "conda list" for the particular virtual environment if that helps.

packages in environment at /home/laytonjb/miniconda3/envs/tf:

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 1.4.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
blas 1.0 mkl
brotli 1.0.9 h5eee18b_7
brotli-bin 1.0.9 h5eee18b_7
ca-certificates 2023.7.22 hbcca054_0 conda-forge
cachetools 5.3.1 pypi_0 pypi
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.2.0 pypi_0 pypi
cloudpickle 2.2.1 pypi_0 pypi
cmake 3.25.0 pypi_0 pypi
contourpy 1.0.5 py39hdb19cb5_0
cudatoolkit 11.8.0 h6a678d5_0
cycler 0.11.0 pyhd3eb1b0_0
dbus 1.13.18 hb2f20db_0
expat 2.4.9 h6a678d5_0
filelock 3.9.0 pypi_0 pypi
flatbuffers 23.5.26 pypi_0 pypi
fontconfig 2.14.1 h52c9d5c_1
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 h4a9f257_0
gast 0.4.0 pypi_0 pypi
giflib 5.2.1 h5eee18b_3
glib 2.69.1 he621ea3_2
google-auth 2.22.0 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.56.2 pypi_0 pypi
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
h5py 3.9.0 pypi_0 pypi
horovod 0.28.1 pypi_0 pypi
icu 58.2 he6710b0_3
idna 3.4 pypi_0 pypi
importlib-metadata 6.8.0 pypi_0 pypi
importlib_resources 5.2.0 pyhd3eb1b0_1
intel-openmp 2023.1.0 hdb19cb5_46305
jinja2 3.1.2 pypi_0 pypi
jpeg 9e h5eee18b_1
keras 2.13.1 pypi_0 pypi
kiwisolver 1.4.4 py39h6a678d5_0
krb5 1.20.1 h143b758_1
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libbrotlicommon 1.0.9 h5eee18b_7
libbrotlidec 1.0.9 h5eee18b_7
libbrotlienc 1.0.9 h5eee18b_7
libclang 16.0.6 pypi_0 pypi
libdeflate 1.17 h5eee18b_0
libedit 3.1.20221030 h5eee18b_0
libevent 2.1.12 hdbd6064_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libllvm10 10.0.1 hbcb73fb_5
libpng 1.6.39 h5eee18b_0
libpq 12.15 hdbd6064_1
libstdcxx-ng 12.3.0 h0f45ef3_0 conda-forge
libtiff 4.5.0 h6a678d5_2
libuuid 1.41.5 h5eee18b_0
libwebp 1.2.4 h11a3e52_1
libwebp-base 1.2.4 h5eee18b_1
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 hfa300c1_0
libxml2 2.9.14 h74e7548_0
libxslt 1.1.35 h4e12654_0
lit 15.0.7 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_0
markdown 3.4.3 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.7.1 py39h06a4308_1
matplotlib-base 3.7.1 py39h417a72b_1
mkl 2023.1.0 h6d00ec8_46342
mkl-service 2.4.0 py39h5eee18b_1
mkl_fft 1.3.6 py39h417a72b_1
mkl_random 1.2.2 py39h417a72b_1
mpmath 1.2.1 pypi_0 pypi
munkres 1.1.4 py_0
nccl 2.12.12.1 h0800d71_0 conda-forge
ncurses 6.4 h6a678d5_0
networkx 3.0 pypi_0 pypi
nspr 4.35 h6a678d5_0
nss 3.89.1 h6a678d5_0
numpy 1.24.3 pypi_0 pypi
numpy-base 1.25.0 py39hb5e798b_0
nvidia-cublas-cu11 2022.4.8 pypi_0 pypi
nvidia-cublas-cu117 11.10.1.25 pypi_0 pypi
nvidia-cublas-cu12 12.2.4.5 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.2.128 pypi_0 pypi
nvidia-cudnn-cu11 8.6.0.163 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 3.0.9 h7f8727e_0
opt-einsum 3.3.0 pypi_0 pypi
packaging 23.1 pypi_0 pypi
pcre 8.45 h295c915_0
pillow 9.3.0 pypi_0 pypi
pip 23.2.1 pypi_0 pypi
ply 3.11 py39h06a4308_0
protobuf 4.23.4 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pyparsing 3.0.9 py39h06a4308_0
pyqt 5.15.7 py39h6a678d5_1
pyqt5-sip 12.11.0 py39h6a678d5_1
python 3.9.17 h955ad1f_0
python-dateutil 2.8.2 pyhd3eb1b0_0
pyyaml 6.0.1 pypi_0 pypi
qt-main 5.15.2 h327a75a_7
qt-webengine 5.15.9 hd2b0992_4
qtwebkit 5.212 h4eab89a_4
readline 8.2 h5eee18b_0
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
setuptools 67.8.0 py39h06a4308_0
sip 6.6.2 py39h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h5eee18b_0
sympy 1.11.1 pypi_0 pypi
tbb 2021.8.0 hdb19cb5_0
tensorboard 2.13.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorflow 2.13.0 pypi_0 pypi
tensorflow-addons 0.21.0 pypi_0 pypi
tensorflow-estimator 2.13.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.32.0 pypi_0 pypi
tensorrt 8.6.1 pypi_0 pypi
tensorrt-bindings 8.6.1 pypi_0 pypi
tensorrt-libs 8.6.1 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
torch 2.0.1+cu118 pypi_0 pypi
torchaudio 2.0.2+cu118 pypi_0 pypi
torchvision 0.15.2+cu118 pypi_0 pypi
tornado 6.2 py39h5eee18b_0
triton 2.0.0 pypi_0 pypi
typeguard 2.13.3 pypi_0 pypi
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2023c h04d1e81_0
urllib3 1.26.16 pypi_0 pypi
werkzeug 2.3.6 pypi_0 pypi
wheel 0.38.4 py39h06a4308_0
wrapt 1.15.0 pypi_0 pypi
xz 5.4.2 h5eee18b_0
zipp 3.16.2 pypi_0 pypi
zlib 1.2.13 h5eee18b_0