When Model Compression Gains Come at the Cost of Output Distribution Collapse
You trimmed a 7B parameter model down to 4 bits. output doubled. Latency halved. But in output, the model started misclassifying rare classe it used to get proper. Accuracy on the held-out probe set dropped only 1.2% — yet customer complaints spiked. This is output distribu collapse: the compressed model's prediction shift away from the original distribual in ways that standard accuracy metric mask. It's not a bug in your code; it's a silent failure of compression. Who Needs This and What Goes off Without It An experienced technician says the trade-off is speed now versus rework later — most shops lose on rework. crews deploying LLMs or vision model on mobile/edge devices You are the engineer who just got a model to fit on a phone—great.