AI's most prominent existential threat: model collapse. If you've heard the term but don’t quite know what it means — don’t worry — we’re going to dive deep into what some consider a technical nightmare (with maybe some hope on the horizon).
What is model collapse, really?
At its core, model collapse is a degradation phenomenon that occurs when AI systems are repeatedly trained on data generated by other AI systems instead of diverse, high-quality human-generated content. It's akin to making a copy of a copy — the fidelity diminishes with each generation, potentially leading to less accurate and reliable models.
Researchers like those at Meta and NYU have repeatedly found that training models on synthetic data can lead to a degradation of performance over time. The synthetic content, no matter how good it seems initially, introduces subtle errors that compound through multiple training cycles. Over time, these models begin to forget what reality looks like and start regurgitating gibberish that loses the diversity and richness required to solve complex tasks.
This recursive use of AI-generated content degrades the model’s perception of reality, effectively causing it to “forget” the tails of distributions — i.e. the rare and nuanced details that make information accurate and useful.
The dangers of model collapse
The implications are dire if left unchecked. If AI models become increasingly reliant on AI-generated data, the errors don’t just multiply, they metastasize, potentially resulting in entire ecosystems of technology built upon distorted, unreliable models. Businesses relying on automated decision-making face decreased accuracy, customer service becomes robotic (in the worst possible way), and forecasts — financial, weather, or otherwise — could become laughably bad.
This is particularly relevant for large language models like ChatGPT and Gemini, which thrive on vast, diverse datasets. The more they feed on content generated by previous versions of themselves, the worse they get at understanding the nuances of human language, making them less helpful and more likely to spread misinformation. It’s like generative AI collectively slipping into a feedback loop of delusion.