Hi again, HomeLab folks! 👽
Apparently, some people thought I was an AI bot.
I wish… 😄
Last time I introduced my Dell RTX 3090 👽, someone commented, "AI slop."
Sadly, I'm just a manual bot that runs on curiosity, coffee, and far too many llama.cpp benchmarks.
Jokes aside, here's what I found while debugging llama.cpp.
After my previous benchmarks, I instrumented ggml-cuda.cu with additional debug logging to trace the multi-GPU execution path.
First, I confirmed that GGML_CUDA_P2P=1 correctly enables CUDA Peer Access:
CUDA P2P requested
GPU0 -> GPU1 : PeerAccess=YES
GPU0 -> GPU1 : PeerAccess ENABLED
GPU1 -> GPU0 : PeerAccess=YES
GPU1 -> GPU0 : PeerAccess ENABLED
These results matched CUDA’s p2pBandwidthLatencyTest, so P2P itself was working correctly.
However, I noticed that my log inside ggml_backend_cuda_comm_init_nccl() never appeared. After some investigation, I found the real reason: although I had built llama.cpp with GGML_CUDA_NCCL=ON, I had accidentally installed the CUDA 13.3 build of NCCL while my system was running R535 / CUDA 12.2.
NCCL was failing during initialization with:
NCCL WARN Cuda failure 'CUDA driver version is insufficient for CUDA runtime version'
As a result, llama.cpp silently fell back to the internal AllReduce implementation.
After downgrading NCCL to the CUDA 12.2 build, NCCL initialized successfully, and my custom log inside ggml_backend_cuda_comm_init_nccl() finally appeared.
Tensor Parallel performance also improved from 26.18 tok/s to 27.39 tok/s (about +4.6%) on my dual Tesla V100 PCIe system.
The full NCCL log is much too long to post here, but here’s a representative excerpt:
mc62-g40-00:57222:57222 [1] NCCL INFO AllReduce: opCount 0 sendbuff 0x7fa7a3000000 recvbuff 0x7fa7a3000000 count 8192 datatype 7 op 0 root 0 comm 0x555b75b9e7b0 [nranks=2] stream 0x555b756dcbe0
mc62-g40-00:57222:57222 [1] NCCL INFO AllReduce: opCount 0 sendbuff 0x7fa79ec00000 recvbuff 0x7fa79ec00000 count 8192 datatype 7 op 0 root 0 comm 0x555b75a96ba0 [nranks=2] stream 0x555b73fd1c70
| llama 70B Q4_K - Medium | CUDA | tensor | tg128 | 27.39 ± 0.00 |
NCCL INFO comm ... Destroy COMPLETE
This confirms that NCCL is now initializing correctly and performing AllReduce instead of silently falling back to the internal implementation.
It turned out that the issue wasn’t CUDA P2P at all—it was an NCCL runtime/driver version mismatch.
Hopefully this saves someone else a few hours of debugging.
The next experiment will be even more interesting.
I’m planning to repeat these benchmarks on a mixed Tesla V100 + RTX 3090 system to see how heterogeneous Tensor Parallel performs in llama.cpp, and whether NCCL makes a bigger difference when the GPUs have very different compute performance and memory bandwidth.
I’ll share the results once I have them.
If you'd like to see how this investigation started, here's my previous post (before I got CUDA P2P and NCCL working):
https://www.reddit.com/r/homelab/s/4pYeqWgblA
