GEN-1.5: Robots Learn New Tasks from a Single Video Demo

Generalist AI has announced GEN-1.5, a model which allows a robot to learn a new task based on a single 3–12 second video demonstration. This model achieves 59% success rate out of the box for 10 different tasks, and 83% success rate after 10 training iterations on 5 minutes of data. The video demonstration is added to the robot’s “context window” as a “physical prompt,” without requiring any retraining of the model.

Previously, robots needed to be trained on hours of specialized data, and then undergo a full finetuning process to learn a new task. In-context learning for robots is analogous to few-shot prompting for language models; if it can be made to work at scale, we’ll be able to teach robots new tasks directly from videos, without the need for ML engineers.