Apple Finds a Theoretical Ceiling for Human-Aligned AI Training
Apple published research on the limits of language model alignment. Best-of-N comes close to the theoretical maximum, while PPO and GRPO fall noticeably short.
Apple AI research has released a paper about how to train language models to be aligned with human values. Alignment here refers to modifying the model’s outputs to better fit our preferences while preserving its capabilities. They derive an upper bound on the potential improvement in the quality of the answer given that the model is restricted to deviate from the initial solution by some constant factor. Best-of-N, where N samples are generated and then the highest scoring sample is selected, almost reaches this bound. On the other hand, both PPO and GRPO, two common reinforcement learning based training algorithms (where the model is trained to give correct answers by providing rewards), perform quite poorly.
PPO and GRPO are used by most organizations that perform finetuning of language models, meaning that this gap directly affects them. Some of the lost quality is due to the choice of training algorithm rather than the amount of data or compute available. Furthermore, they demonstrate that averaging multiple reward scores helps prevent reward hacking, which occurs when the model starts optimizing for the reward function itself instead of actually solving the task at hand.
We don’t know if this upper bound generalizes to larger models than those considered in the paper. We also don’t know if this will affect the way OpenAI, Google, or Anthropic train their models since none of them disclose sufficient information about their training procedures.
Source: machinelearning.apple.com
Free course
Stop reading about AI — start building with it
The free Claude Code course: your first site, tool or game — no coding. No upsells, no cross-sells — nothing to buy here.
Start free →
Author
Evgenii Arsentev
PhD · Founder & CEO, Dad4Kids
Articles · Latest articles