MiniMax's MSA Slashes Long-Context Compute 28x

MiniMax released MSA — short for MiniMax Sparse Attention — a method that cuts the per-token compute of attention by 28.4x at a one-million-token context while, the company says, matching ordinary full attention on standard benchmarks. It's not a paper-only experiment: MiniMax has already shipped it inside its production model, MiniMax-M3, and open-sourced the inference kernel under an MIT license.

Here's the bottleneck it attacks. The "attention" mechanism at the heart of every modern language model compares each token to every other token, so the cost grows with the square of the input length. Feed in a whole codebase or a book — the long contexts everyone now wants — and that quadratic cost is what makes it slow and expensive. Sparse attention tries to skip the comparisons that don't matter, and the trick is doing that without making the model dumber.

1How the two-branch trick works

MSA splits the job in two. A lightweight "Index Branch" first scores chunks of the input — blocks of 128 tokens — and picks the top 16 most relevant ones for each query, always keeping the local block the query sits in. Then the "Main Branch" does the real, exact attention math, but only over those selected blocks instead of the entire input. So each token attends to a fixed budget of about 2,048 tokens no matter how long the document is, which is why the savings explode at long context: 28.4x less compute per token at 1M, and on Nvidia H800 GPUs, 14.2x faster prefill and 7.6x faster decoding.

The number that matters more than speed is whether quality holds. MiniMax reports its from-scratch MSA model essentially ties full attention — MMLU 67.2 vs 67.0, GSM8K 77.7 vs 76.2, HumanEval 64.0 vs 61.0 — and actually edges ahead on a long-context retrieval test (RULER-8K, 84.2 vs 79.8). In other words, they claim you get the big speedup roughly for free. The base model is a 109B-parameter Mixture-of-Experts trained on 3 trillion tokens, and the released kernel, fmha_sm100, supports modern low-precision formats like FP8 and FP4.

Why care if you're not training models? Because this is the kind of plumbing improvement that quietly makes the tools you use cheaper and faster. Long context is what lets an AI read your entire project, a long PDF, or months of chat history at once — and right now that's the expensive setting. Make it 28x cheaper to run and it stops being a premium feature and starts being the default. The honest caveat: every lab claims its sparse-attention scheme matches full attention, and the gaps usually show up on the hard, weird inputs the benchmarks miss. I'd treat the "free lunch" framing as "cheap lunch" until someone outside MiniMax reproduces it.

There are real limits in the fine print. The open kernel only targets Nvidia's newest SM100 GPUs for now; other hardware needs its own implementation. Some benchmark subtasks still show small gaps versus full attention, and all the results come from MiniMax's own evaluation suite with no third-party reproduction yet. Still, shipping it in a production model rather than a demo is the meaningful signal — the efficiency race for long context is clearly heating up.