NVIDIA Releases Open Speech Model Nemotron 3.5 Transcribe

NVIDIA introduced their open-weight speech recognition model, Nemotron 3.5 Transcribe. This model has 30 billion parameters, however, it only uses around 4 billion of them at any given time due to its mixture-of-experts architecture which allows it to use only a fraction of the model for each task. In one checkpoint (model weight file), it can perform audio transcription, speaker identification, and language translation. On the Hugging Face Open ASR Leaderboard, a benchmark for open speech recognition models, it achieved an error rate of 5.33%, which is the lowest among all open models. By fine-tuning it with some domain specific data and training it further on medical data from Eka Care (a medical records company), they were able to reduce the error rate from 22.44% to 12.17%, which is nearly a 46% improvement!

This is a huge leap forward for voice agents, where you have software that can listen and speak back to you. Instead of having to integrate multiple APIs to get audio transcribed, identified who said what, and translated into other languages, you can now do everything with one model. Also, since the weights are open source, you can deploy this model locally on your own GPU (graphics processing unit, used for artificial intelligence) and process 100 hours of audio data within 3 minutes, without having to constantly call some external API. Lastly, the fact that they were able to reduce the error rate by almost 46% when fine tuning on medical data demonstrates how well this model can generalize to specialized domains rather than simply performing well on generic benchmarks.