Kimi K3.1 Pro Drops From 4th to 9th in New Artificial Analysis Index

The Intelligence Index v5 has been published by Artificial Analysis, which provides AI model scoring and ranking. This update includes a major shift in the methodology, where 60% of the score is now based on private test data (questions the AI hasn’t seen before). This aims to curb “benchmaxxing”, or optimizing your model specifically to perform well on specific types of questions asked in a particular benchmark. Following this change, the difference between the best open source models and the best proprietary models increased from 3 points to 12 points. Open source model Kimi K3.1 Pro, developed by Chinese research lab Moonshot AI dropped from 4th place to 9th place, and is now outperformed by 8 different proprietary models. Similarly, xAI’s model Grok 4.7 dropped by 12 points.

This decrease in performance for Kimi is consistent with a pattern already visible to anyone running these models on real agent tasks — Chinese open source models tend to perform well on public benchmarks where the questions are known ahead of time, but struggle when faced with unknown questions. Using private test data helps prevent benchmaxxing for all models, both open and proprietary, providing a fairer comparison across the industry.