Good training helps make AIs more robust against attack

A new paper from OpenAI demonstrates how injecting a tiny bit of “beneficial trait” training into a model’s usual post-training reinforcement learning process can yield significant safety improvements. The authors found that when they added a small fraction of “beneficial trait” examples (e.g., truth-telling, expressing uncertainty, corrigibility, transparency, equity, human welfare) across various domains such as medicine, education, science, law, and engineering into the typical training process, they were able to produce models that performed significantly better on 44 out of 53 different benchmarks for deception, honesty, suckuppery, and reward hacking. This work was recently profiled in The Decoder, and is exciting since it shows that these improvements generalize widely across a variety of scenarios, even though the training data itself was limited to only a few specific scenarios.

This approach is remarkably simple. To do this, the authors crafted a collection of plausible dialogues that encouraged certain beneficial traits. They then added a small fraction of these examples into the regular training process. There was nothing fancy going on here — they simply injected some good habits into the training data.

1The part that surprised me

One cool thing about this approach is that it appears to work across domains. For example, training a model to exhibit good behavior in medical contexts seemed to improve its performance on other types of questions, and vice versa. Essentially, good behavior seems to be contagious. This is similar to previous findings which showed that training a model to exhibit bad behavior in a particular context could lead to widespread problems, but now we see that good behavior can be contagious too!

These models were also robust to adversarial attacks. According to the authors, “adversarial prompts had far less effect” compared to the baseline model, and they were able to resist attempts at harmful fine tuning while continuing to follow valid instructions (which they refer to as “selective persistence”). Basically, these models were more difficult to hack into saying something bad, but they were still willing to follow your instructions.

For most people, this is important because much of the danger from AI comes not from it trying to be evil, but rather from it being naive — believing whatever you tell it, being manipulated by adversarial inputs, or boldly asserting false claims. By showing that a small amount of “be honest, admit when you don’t know something, let me correct you” training generalizes across the entire model, this provides a straightforward way to reduce these kinds of errors. OpenAI defines these traits as “empirical behaviors” whereas Anthropic writes them out explicitly as a constitution, but they are really just two sides of the same coin. My main point here is that it seems like improving AI safety in this case didn’t require any heroic innovations, but rather just planting some good seeds that grow on their own.