Small BAR Size Support?
Qubitium opened this issue · comments
NVIDIA Open GPU Kernel Modules Version
550.90.07
Operating System and Version
Ubuntu 22.04
Kernel Release
6.8.9
Hardware: GPU
4090
Describe the bug
We have tested the modified kernel on two systems: 1x intel desktop (with full BAR=32GB), 1x amd server (without resizable_bar, BAR=256MB,512MB)
On the intel with full 32GB BAR size for the 2x4090, NCCL/P2P test is passing with modded driver.
However on the amd server platform where bios doesn't support resizable_bar, nvidia-smi is only showing 256MB and 512MB bar sizes for the 2x4090. On this amd server, even with the this modded nvidia driver, NCCL/P2P tests failed. The amd server also has lots of pcie devices so it may be running out of pcie map space to assign the large 32GB bars that 4090 support.
So my question is, is the current P2P+4090 code only working if BAR size >= full 4090 GPU vram size? Thank you!
same question here =)
What motherboard is AMD? I just updated bios on mine and it got Re-Size Bar support
same question here, here is our configuration
dual Epyc 7742, supermicro H12DSG system
here is the lspci information
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] GA102 [GeForce RTX 3090]
Flags: bus master, fast devsel, latency 0, IRQ 402, NUMA node 0
Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Memory at 40090000000 (64-bit, prefetchable) [size=256M]
Memory at 400a0000000 (64-bit, prefetchable) [size=32M]
I/O ports at 1000 [size=128]
Expansion ROM at f7000000 [virtual] [disabled] [size=512K]
Capabilities:
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
here is the error information from simpeP2P test
Enabling peer access between GPU0 and GPU1...
CUDA error at simpleP2P.cu:129 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"