Why do transformers have outliers?

By Anwaar Khalid in Quantization on

Modern Machine Learning models are trained with a large number of parameters, often too large, and this overparameterization is very useful during training as it creates a vast search space for the model to encode rich representations from data...

Integer Quantization: Deep Dive 🤿

By Anwaar Khalid in Quantization on

A lot has happened in transformer quantization over the past few years, from barely being able to quantize a 7B model in INT8 without destroying accuracy, to routinely fitting a 70B model in 4-bits on a single GPU. But existing guides on the...