ByteDance developing a 10T parameter pre-trained model

According to the Financial Times, ByteDance is building a massive language model with up to 10 trillion parameters. The model is being pre-trained right now (which takes anywhere from 3–6 months) and is being developed by their 2,000 person Seed team.

This model is going to be three times larger than the current largest Chinese language model (Kimi K3 by Moonshot), and probably going to perform similarly to Anthropic’s Mythos 5 (estimated to have ~8 trillion params). Additionally, xAI is currently pre-training their Grok model at 6 and 10 trillion parameters on their Colossus 2 cluster (as reported by Elon Musk).

This team has been tasked by founder Zhang Yiming with “building world leadership in long-horizon model capabilities”. They have been abstaining from distillation (training smaller models on the output of larger ones) for more than a year now, instead focusing on building out their own pre-training methods.

While there are many factors that go into making a high quality language model, it’s clear that tech companies view large models as very valuable assets. They see potential in large architectures that hasn’t yet been fully realized, and whoever can unlock this potential first will gain a significant competitive edge.