MLP-only vs. MLP + attention quantization
Keeping attention in BF16 cut KL by 49 to 54% across all four recipes
MLP + attention quantized
MLP only quantized
MLP only + FP8 KV cache
MLP-only vs. MLP + attention quantization
8-BIT WEIGHTS
0.000
0.001
0.002
0.003
0.004
W8A16
0.001898
0.000970
W8A8
0.003670
0.001849
0.002442
4-BIT WEIGHTS
0.00
0.01
0.02
0.03
0.04
W4A16
0.01939
0.008874
W4A4 NVFP4
0.03485
0.01652
0.01689
Forward KL against the original model (nats) · lower is better