QAD held-out KL and task reward
The lowest-KL checkpoint was not the highest-reward checkpoint
Normal QAD · top-500 KL
Deployment-informed · top-500 KL
Normal QAD · one-hot CE
Deployment-informed · one-hot CE
NVFP4 · no QAD
BF16 mean reward
QAD held-out KL and task reward
0.30
0.32
0.34
0.36
0.38
0.40
0.42
0.44
0.0175
0.0200
0.0225
0.0250
0.0275
0.0300
0.0325
0.0350
0.0375
Mean task reward
Held-out forward KL from the original BF16 model (nats)
BF16 mean reward 0.371