A good news, we can enable P2P for Dual RTX 4090 or more. When running simpleP2P script, we will got this results. Don’t worry, next is how to enable it easily!
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2
Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : No
> Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA GeForce RTX 4090 (GPU0) : No
Two or more GPUs with Peer-to-Peer access capability are required for ./simpleP2P.
Peer to Peer access is not available amongst GPUs in the system, waiving test.
First thing, make sure to enable Resize BAR and disable IOMMU in BIOS (I’m using ASUS WRX80SAGE).
Next, uninstall all nvidia drivers (*yes, thats right!)
# Uninstall all nvidia
sudo apt-get --purge remove "*nvidia*"
sudo apt-get --purge remove "*cuda*" "*cudnn*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" "*libnccl*"
# disable iommu
ll /sys/class/iommu/
# install dependencies
sudo apt install git cmake
# reboot
sudo rebootNow, we will be fallback to Nouveau driver. Its okay. Here is the steps to disable or you can google it.
sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo update-initramfs -u
sudo rebootNext, download the NVIDIA Driver and install it
wget -c https://us.download.nvidia.com/XFree86/Linux-x86_64/565.57.01/NVIDIA-Linux-x86_64-565.57.01.run
# Choose the MIT/GPL, don't proprietary
sudo sh ./NVIDIA-Linux-x86_64-565.57.01.run --no-kernel-modulesThen install tiny P2P
git clone git@github.com:tinygrad/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules
./install.sh
sudo depmod
sudo rebootNext, install cuda samples and test!
git clone https://github.com/NVIDIA/cuda-samples.git
rm -rf bin
make
./bin/x86_64/linux/release/deviceQuery
Voilla! Now you see P2P Enabled!
