GitHub radar

$0.0007/task for evaluating AI reasoning skills with ARC Task Gen

Pathway’s ARC Task Gen creates new ARC-AGI-1-like reasoning tasks, alongside a paper demonstrating that a 150M parameter model outperforms its much larger competitors at 11x the efficiency.

01pathwaycom/arc-task-gen 3.8kPython

The ARC Task Gen tool generates ARC-AGI-1-like reasoning tasks. These tasks are distribution matched to the public eval data. This repository accompanies our paper describing our novel model architecture, BDH-CQ, that includes iterative reasoning and in-context learning via evolving recurrent memories in a continuous latent space. A 150M parameter version of our BDH-CQ model gets a pass@2 score of 29.5% on the ARC-AGI-1 eval set at only $0.0007/task, making it 11x more efficient than GPT-5.6 Luna Low. Our synthetic tasks use the exact same ARC JSON format as real ARC tasks, meaning they can be evaluated using the same tools.

Why a vibe-coder should care

In this paper we show that a much smaller model (10–100x smaller than SOTA) can match performance on structured reasoning tasks while being significantly more efficient. Want to see what the future of efficient reasoning looks like? Or how to efficiently reason? Check out our work!

How to install

Copy this and send it to your agent — Claude Code, Codex, any of them:

Set up https://github.com/pathwaycom/arc-task-gen (pixi install), ask me for an API key, and generate a tasks.json set per instructions.md.

Runs on a regular laptop — only Python needed.

Open on GitHub