01
Batching and kernel selection
02
Worker assignment and chunking
03
Operand format and compute precision
04
Casts and fusion boundaries
Batch shape selects a different kernel and reduction order.
TRAINING ACTOR · BATCHED
Many rows
Same weights
Batched GEMM
Work spread over rows
Accumulate → round
One kernel’s reduction order
ROLLOUT ENGINE · SINGLE ROW
One row
Same weights
K1
K2
K3
Reduce partial sums
A different reduction order
Rounding partial sums changes the combined result.
TRAINING ACTOR · ONE DOT
Sum 6,144 products
Higher-precision accumulation
Round final sum
BF16 output
ROLLOUT ENGINE · TWO SHARDS
GPU 0 · 3,072
Partial sum u
GPU 1 · 3,072
Partial sum v
Round u
BF16 partial
Round v
BF16 partial
Add rounded partials
Then produce the output
BF16 and FP8 instructions group the same products differently.
TRAINING ACTOR · QDQ
FP8 → BF16
Values preserved
Two groups of 16
Inspected BF16 instructions
FP32 accumulator
Updated twice
ROLLOUT ENGINE · NATIVE FP8
FP8 × FP8
Values preserved
One group of 32
Inspected FP8 instruction
FP32 accumulator
Updated once
Fusing operations can remove an intermediate rounding step.
TRAINING ACTOR · MATERIALIZED
FP32 sum
r + u
BF16 cast
Round here
RMSNorm
Rounded input
BF16 output
Final cast
ROLLOUT ENGINE · FUSED
FP32 sum
r + u
No cast here
RMSNorm
Unrounded input
BF16 output
Final cast