By A project called ATLAS V3 posted 74.6% on LiveCodeBench v5 this week using a frozen 14B parameter model running on a single RTX 5060 Ti, a GPU that costs around $500. Claude 4.5 Sonnet scores 71.4% on the same benchmark. The Hacker News post titled "$500 GPU outperforms Claude Sonnet on coding benchmarks" climbed to the front page and racked up hundreds of points and comments. The replies were exactly what you would expect: genuine enthusiasm alongside people who spotted the methodology caveat buried in the README.
Both reactions are right.
What ATLAS V3 actually does
ATLAS stands for Adaptive Test-time Learning and Autonomous Specialization. It is an inference-time scaffolding pipeline wrapped around Qwen3-14B-Q4_K_M, a quantized model that never gets fine-tuned. No gradient updates. No training data. The weights are frozen from the start.
The V3 pipeline runs in three phases. Phase 1 generates multiple candidate solutions using PlanSearch, BudgetForcing, and diversified sampling. Phase 2 routes those candidates through a Lens selection layer. Phase 3 runs self-verified iterative repair: the model generates its own test cases, checks its solution against them, and applies PR-CoT (program repair chain-of-thought) to fix failures.
The ablation study tells the real story. Qwen3-14B without any scaffolding sits at 54.9%. Add Phase 1 and it jumps to 67.3% — a 12.4 percentage point gain. Phase 2 contributes exactly zero. Phase 3, the self-verified repair loop, pushes it to 74.6%, adding another 7.3 points. PR-CoT rescues 36 out of 42 tasks that failed initially, an 85.7% rescue rate. All the meaningful gains come from generation diversity and iterative repair. The Lens routing layer in the middle does nothing on its own.
No API calls. No cloud. The model generates its own verification tests and never sees the answer key during repair. One box, one GPU.
The comparison is not clean, and the project says so
To the project's credit, the README is explicit about the methodology. The 74.6% figure uses "pass@1-v(k=3)," defined as: one solution submitted per task, generated via best-of-3 candidates, Lens selection, and iterative repair. It is not standard pass@1, which means single-shot generation at temperature 0.
The competitor numbers come from Artificial Analysis (https://artificialanalysis.ai/evaluations/livecodebench) and use single-shot pass@1 on 315 problems. ATLAS ran on 599 problems. Different task sets, different methodology, different generation budgets. The README itself states this is not a controlled head-to-head.
So the headline comparison is technically misleading. That is also true of roughly half the benchmark posts circulating at any given moment. At least this project flags the issue in its own documentation rather than burying it.
