pipette
ESEspañol

Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman, Pashmina Cameron

PreprintReal-world use

In the authors' words

Post-training quantization (PTQ) compresses the weights of a trained model with limited compute and data. Down to 4 bits it is near-lossless, but at 2 bits scalar PTQ configurations fall to chance-level accuracy. We study three choices usually treated as implementation details that matter little at 4 bits but decide 2-bit quality: the quantization representation, the calibration data, and the recovery fine-tuning data, across six dense and mixture-of experts models (ranging from to B parameters). Representation is the largest factor and vector codebooks restore usable models where scalar PTQ pipelines collapse. Most of the remaining loss in accuracy is on generative tasks, and the amount of performance drop is model-dependent. Furthermore, at a short recovery budget, calibration and recovery data are coupled. The best recovery data depends on the calibration data, so the two must be chosen jointly. The resulting recipe involves a vector codebook representation, a small task-aligned calibration mixture, and an inexpensive codebook-only recovery stage. Including codebooks, it uses about effective bits per weight and retains to of full-precision question-answering (QA) accuracy where scalar PTQ is at chance, and, including recovery, uses to less compute than the strongest quantization-aware-training (QAT) baseline.

Main resultLimitation the authors admit

Appeared: Wednesday, October 21. arXiv. Preprint, not yet peer-reviewed.