Why Are H100s Idle in 2026? And What’s Replacing Them?

While we debate over the abilities of these new AI models, it’s important to know why Nvidia H100s (the most commonly used chip for AI applications) are often sitting idle (50–80% of the time) because of their memory constraints. These new chips aim to solve this problem differently. For instance, the Cerebras WSE-3 wafer-scale chip (the size of a dinner plate) with 44GB of on-chip memory is able to process 1000+ tokens/sec. Meanwhile, Etched Sohu is able to run Llama 70B at 500,000 tokens/sec.

Inference speed and cost have an impact on what types of agents we can build today (and what we still can’t build). We’ve seen Nvidia release their NVFP4 (a 4-bit compression format that offers 3x speedup with less than 1% accuracy loss). It’s difficult to predict which architecture will come out on top, but each new chip brings us closer to unlocking the potential of AI for developers that are currently leveraging it.