GitHub radar
Auto-BenchMax: Improving 2x On Tool-Use Benchmark Scores
A open source training data generation pipeline that doubles your LLM’s score on tool-use/agency benchmarks. One iteration improved Qwen3-Coder-30B from 19.1 to 40.4 on the MCP-Atlas benchmark (over 2x improvement)!
This is an open source pipeline developed by community member ‘sunny-glow’ to generate training data that doubles your LLM’s scores on tool-use/agency benchmarks. You can find the full pipeline, training code (Qwen3-Coder-30B trained using axolotl + DeepSpeed ZeRO-3), and first iteration synthetic dataset (1,312 examples) in this repo. Using this pipeline, we were able to improve our score on the MCP-Atlas public benchmark from 19.1 to 40.4 after one iteration. This pipeline supports both Execute-then-Extract for rule-based benchmarks and Construct-then-Verify for LLM evaluated benchmarks. You can even use this pipeline with just one instruction from Claude Code using the provided skill!
Why a vibe-coder should care
If you are looking to fine tune your models for agency or tool use, this repo provides a clear, replicable way to achieve 2x improvements on any benchmark, while also making everything open source for the community.
▌ More finds