Poolside Laguna: A small model that outperforms trillion parameter models

Poolside just published their new model, Laguna S 2.1. It’s an open source code generation model with 118 billion parameters, but only 8 billion are used at any one time. It performed better than trillion parameter models on the DeepSWE benchmark for AI developers (with a score of 40.4%) and even solved Erdős Problem #397 (which has been unsolved since 1975) in 40 steps for only $0.09!

This was achieved by training the model to be “persistent”, i.e., to double check its work, try different strategies if something doesn’t work out, etc. So it turns out that how you train your model is just as important as how big it is. There are many ways to win besides racing towards a trillion params.