🤖 Self-correcting AI? It's not science fiction - it could be the future! 💡
Modern large language models (LLMs) still struggle to correct their own mistakes, especially when no external help is available. Traditional models struggle to self-correct, and existing approaches to teaching LLMs to self-correct often rely on external models or supervised fine-tuning (SFT) based on correction examples. These approaches face challenges such as distribution mismatch, where corrections in training differ from those generated by the model at test time, or are limited to making only minimal changes to avoid degrading correct responses.
That's why researchers at Google DeepMind have just introduced SCoRe (Self-Correction via Reinforcement Learning) - an innovative, multi-turn reinforcement learning approach that finally helps AI self-correct using only self-generated data!
Stage 1 is when the LLM trains on self-generated correction data. Of course, some regularisation has to be applied here to prevent overfitting to small corrections by constraining the first attempt responses to be close to the original model.
In stage 2, the model undergoes multi-turn reinforcement learning with a reward shaping strategy that incentivises self-correction. A bonus is applied to corrections that shift incorrect responses to correct ones, minimising the likelihood of degrading correct responses.
SCoRe's two-stage training using these self-generated responses to improve accuracy over time achieved impressive gains in maths (+15.6%) and coding (+9.1%) without relying on multiple models or external feedback.
This brings us one step closer to our original goal of a model that can detect and correct its own errors, pushing the boundaries of AI autonomy.
⚠️ But there are still challenges to overcome: SCoRe is resource-intensive and currently limited to single-round corrections. Nevertheless, this breakthrough shows that AI can be trained to reliably improve itself, moving us towards models that can perform complex tasks independently.
Exciting times for AI! 🌍✨
And for those of you who would like to read the news in full: https://lnkd.in/ektyhWnZ
#AI #MachineLearning #ReinforcementLearning #AIBreakthroughs #Innovation #TechFuture #SelfCorrection #AIResearch
Image created with DALL-E using ChatGPT 4o.