RoCEv2 (beta)
RoCEv2 generates RDMA over Converged Ethernet v2 traffic. The scenario is designed to test data center networks, storage systems, HPC/AI clusters, and fabrics that support lossless Ethernet.
Table 180 — RoCEv2 settings
| Name | Description | Value |
|---|---|---|
| RDMA command | RDMA operation type | Select a command, such as write |
| Queue Pair count | Number of concurrent RDMA queues | Set the number of QPs; default: 1000 |
| InfiniBand MTU | MTU for RoCEv2 exchanges | Select a value that matches the test environment |
| Initial rate | Initial transmission rate | Used to ramp the load up smoothly |
| Rate increment | Rate change over time | Set the ramp-up step when the load should increase gradually |
| CNP | Generate Congestion Notification Packets | Enable to test how the network responds to congestion notifications |
| CNP interval | Interval between CNPs | Specified in microseconds |
| PFC | Priority Flow Control | Enable to test lossless traffic classes |
| PFC class | PFC priority class | Specify a class, such as 3 |
| DCQCN | DCQCN congestion control | Enable to test behavior under congestion |
| DCQCN settings | G, AI Rate, Alpha, Recovery, Byte Counter | Configure according to the data center profile |
| Payload | Payload size and content | Define the data profile |
| Duration | RoCEv2 traffic generation time | Specify the test duration |
PFC on the agent NIC
When PFC is enabled in a RoCEv2 entry, the agent configures the NVIDIA ConnectX NIC for the duration of the task so that switch pauses actually stop transmission:
- it enables PFC for the selected PFC class;
- it switches the NIC to DSCP trust mode and maps the DSCP of the entry (from IP header (L3), DSCP 0 by default) to the PFC class, so that RoCEv2 frames leave through the queue of this class;
- after the task, it restores the original NIC settings. If the task ended abnormally, the original settings are restored at the next run on this port.
Changing the PFC settings restarts the link of a ConnectX port for about 10 s. The agent waits for a stable link before it starts transmitting and before it finishes the task, so starting and finishing a task with PFC takes that much longer. On the switch, the same class must be configured as lossless (no-drop) on the ingress and egress ports of the path.