# AI Fabric Operations (InfiniBand / RoCE)

Source: https://savrn.com/institute/courses/aio-2902
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

Advanced · AIO

Proposed course specification. Delivery, enrollment, and assessment arrangements require a partner program. Credential references describe intended alignment or pathways; external credentials and academic awards are issued only by the responsible organizations.

120Total hours

32Lecture

88Hands-on lab

AIO 2901Prerequisite

—Proposed credential / alignment

88 lab hours 32 lecture hours

This course covers the scale-out network fabric that connects GPU nodes into a training or inference cluster, covering both InfiniBand and RDMA over Converged Ethernet (RoCE) fabric technologies as alternative implementations of the same lossless, low-latency transport requirement. Students configure and query an InfiniBand subnet manager, use NVIDIA UFM (Unified Fabric Manager) to monitor fabric health and congestion, and diagnose link and port errors. On the Ethernet side, students configure Priority Flow Control (IEEE 802.1Qbb) and Explicit Congestion Notification/DCQCN to build a lossless RoCE fabric, and verify GPUDirect RDMA operation between GPUs across the fabric without CPU-memory bounce buffers. The course covers non-blocking leaf-spine fabric topology at teaching scale, cable and transceiver verification, and NCCL-based fabric bandwidth testing across multiple nodes as the acceptance criterion for a commissioned fabric segment. Students troubleshoot induced congestion and packet-loss scenarios and document root cause using fabric telemetry.

## What you'll be able to do

1. Configure an InfiniBand subnet manager to discover and initialize a multi-switch fabric topology.
2. Use NVIDIA UFM to monitor fabric health and identify a degraded or congested link.
3. Configure Priority Flow Control (IEEE 802.1Qbb) on a lossless RoCE fabric to prevent buffer-overflow packet loss.
4. Configure DCQCN/ECN marking thresholds on switch buffers to provide early congestion signaling for RoCE traffic.
5. Verify GPUDirect RDMA operation between GPUs on different nodes across the fabric without a CPU bounce buffer.
6. Run a multi-node NCCL bandwidth test across the commissioned fabric segment as a fabric acceptance criterion.
7. Verify cable and transceiver integrity for a fabric link using vendor diagnostic counters.
8. Diagnose an induced packet-loss or congestion event on the lab fabric and document root cause using fabric telemetry.
9. Design a non-blocking leaf-spine fabric topology for a specified number of GPU nodes at teaching scale.
10. Distinguish InfiniBand and RoCE fabric architectures by mechanism and select the appropriate technology for a stated site constraint.
11. Document a fabric commissioning report including topology diagram, subnet manager or PFC/ECN configuration, and NCCL acceptance results.
12. Escalate a fabric hardware failure requiring vendor replacement, citing the specific failed component and diagnostic evidence.

← Previous course

GPU Cluster Bring-Up & Commissioning

Next course →

Scheduler & Tenancy Operations (Slurm, Kubernetes, MIG)

### Department

Applied AI & Compute Operations

9 courses · 860 hours →

## Train the team that runs the factory.

Shape a program around your facility, your learners, and the competencies employers need.

[Plan a Program →](https://savrn.com/institute/plan) [Explore programs](https://savrn.com/institute/programs) [Read catalog](https://savrn.com/institute/catalog)