Mistral AI teaches robots how to move around without LiDAR — just cameras

The Mistral team has published the Robostral Navigate model, an 8B parameter model for robot navigation in unseen environments, which can be done with only a standard RGB camera (no need for LiDARs nor depth sensors), making robotics cheaper. On the standard R2R-CE benchmark in unseen environments, it reaches a success rate of 76.6%, 9.7 points higher than any other model using only one camera, and 4.5 points higher than models using depth sensors.

Robostral Navigate is a textual instruction + image input model that predicts the coordinates of the next target position in the view. It was trained in simulation on 400,000 trajectories in 6,000 different environments. It can be used for any type of robot (wheeled, legged, aerial), regardless of its kinematic constraints. Plus, thanks to the prefix caching technique, it reduces the token cost during training by a factor of 22!

So far, robots needed expensive sensors (LiDARs, which can cost thousands of dollars, or stereo cameras) to navigate accurately. With this single lens, a language model can do better, potentially reducing the price of industrial robots by an order of magnitude, for example for logistics, deliveries, or production. Mistral is the first European laboratory to venture into the real world!