5 Startups Are Developing Alternatives to the Transformer Architecture

Every language model since 2017 has used the same architecture: the transformer, described in the “Attention Is All You Need” paper. The dense attention mechanism makes it so that a 10,000 word document takes 50 million multiplications to process, and compute costs grow rapidly as context length increases.

1Five Companies, Five Approaches

MIT Technology Review has written about 5 companies using different math. Inception (Palo Alto) uses diffusion, where the model outputs chunks of text all at once. They claim their Mercury 2 model is equivalent to GPT-4 but is 10x faster. Liquid AI (Cambridge) uses 20% transformers and 80% liquid neural nets to build models that are 4x better than their size on a $50 Raspberry Pi (which has been downloaded 34 million times). Pathway’s Dragon Hatchling was able to solve 97% of 250,000 Sudoku puzzles that other LLMs couldn’t solve.

2Why This Has Become Urgent

OpenAI will spend $50B on compute in 2026, and data center electricity usage will double by 2030. The transformer architecture is reaching physical limits of scalability: we can make models more capable, but it will come at a greater and greater expense. Even if just one of these new architectures works as advertised, future AI models might be much cheaper without actually being any smarter — because the underlying mathematics has changed.