Website Shows Who Was Learned By AI Models

Former OpenAI employees Joey Flynn and Thomas Dimson have developed a website that measures how well AI remembers people based on what it learned during training. Called In the Weights, the site asks various models a question and outputs a “strength score” which indicates how embedded a particular name is within the model’s weights.

The highest possible strength score is 996, and as you might imagine, the top of the list is populated by famous people such as Mozart, Shakespeare, and Taylor Swift, who all achieve maximum strength scores. On the other hand, less well-known individuals receive much lower scores (we checked two members of our team and found scores of 175 and 262). This gives us a tangible metric for understanding how well AI remembers us.

1Why this matters

It’s important to distinguish between being “in the weights” and being “on the internet.” If an AI model can remember you without needing to look you up, then it has effectively memorized you, since your name appeared often enough in the training data that the model decided to remember it. This can have significant implications. It affects how well a chatbot can explain you, or how likely it is to invent false information about you. It also tells us something about which people’s data was prioritized by AI developers, and which ones were left out. If you’re a public figure, journalist, or simply someone with an online presence, knowing your position in the weights provides some insight into how major AI systems will portray you by default.

Of course, there are caveats. Larger models are easier to interpret. Hallucinations are possible, misspellings reduce the strength score, and frequently occurring names generally perform poorly due to dilution. Therefore, a low strength score doesn’t necessarily mean you don’t exist in the data, nor does a high score indicate that the AI knows accurate information about you.

Despite these limitations, In the Weights offers value through its ability to demystify training data, an area that has historically been shrouded in secrecy. Companies typically refuse to reveal what data they used to train their models, while end-users lack the means to investigate. By providing a straightforward way to check whether or not a model remembers you, In the Weights takes a modest yet honest step towards transparency. Next time a chatbot talks about you, remember that it might be repeating what it heard before, or making stuff up.