KL penalty stabilizes async prod RL
One-round async score centering · top-p 0.95 · top-k 20 · three random seeds each
No KL penalty
KL penalty 0.02
0.2
0.3
0.4
0.5
0.6
0.7
0
48
96
144
192
Optimizer steps
Reward