In my last post, I covered Optimal Brain Quantizer (OBQ). OBQ breaks quantization into per-layer, row-wise subproblems: at each step it quantizes the weight that adds the least error, then compensates the remaining unquantized weights using a...
In Part 1, we looked at why rounding direction matters: a stochastic rounding experiment on a single ResNet-18 layer found assignments that beat round-to-nearest by over 10 points, just by choosing up vs. down more carefully. Then we looked into...
Adaptive rounding techniques are one of the best ways to get more accuracy from quantization with the same bitwidth. In this series we’ll cover the evolution of adaptive rounding techniques starting from Adaround which delivers excellent quality...
In my previous blogpost, we discussed in detail the outlier problem in transformers. We realized that outliers are not just random noise - they encode important directional information that lets the model selectively skip updating token...
Modern Machine Learning models are trained with a large number of parameters, often too large, and this overparameterization is very useful during training as it creates a vast search space for the model to encode rich representations from data...
A lot has happened in transformer quantization over the past few years, from barely being able to quantize a 7B model in INT8 without destroying accuracy, to routinely fitting a 70B model in 4-bits on a single GPU. But existing guides on the...