Integer Quantization: Deep Dive 🤿

By Anwaar Khalid in Quantization on

A lot has happened in transformer quantization over the past few years, from barely being able to quantize a 7B model in INT8 without destroying accuracy, to routinely fitting a 70B model in 4-bits on a single GPU. But existing guides on the...