Open Models Are Closing the Gap With Closed AI, Fast

In December 2024, the best open-weight coding model you could run on a single consumer GPU scored 20% on SWE-bench Verified. Eighteen months later it scores 77%. That red line, open models under 35B parameters, is the steepest line on the board. Steeper than Anthropic's. Steeper than OpenAI's. Steeper than Google's.

SWE-bench Verified over time: six model classes plotted by release date, with the open sub-35B frontier (red) rising faster than every closed lab's line

This is the headline chart from the live botlab.dev open-source LLM benchmark dashboard, which refreshes daily against benchlm.ai, SWE-rebench, and BFCL. Every figure in this article is clickable: it links straight to its live, interactive version on the dashboard, so you can hover the points and check the data yourself. Every number below is computed from that dataset.

The claim of this piece is narrow and defensible: the best open sub-35B models are improving faster than the best closed frontier models, and the gap between them is shrinking. Not "open has won." Not "open wins by December." Just: the rate of improvement favors open, the direction is closing, and on the one benchmark that can't be gamed the gap is already down to a few points. Here is the evidence, with the statistics done properly rather than eyeballed.

1. Open is improving ~1.5x faster than closed

Take the running-best model in each group (the frontier, the best thing you could actually download or call at each point in time) and fit a line to it. That is the honest object to compare, because nobody deploys the median model; they deploy the best available one.

  • Open sub-35B frontier: +39 points per year.
  • Closed frontier (Anthropic, OpenAI, Google, xAI combined): +26 points per year.

Open is climbing 1.5x faster, and the difference is not noise: the slope gap is 13.5 points/year with a standard error of 3.6, which is a z of 3.7 and a p-value of 0.0002. You would see a gap this large by chance roughly one time in five thousand.

SWE-bench Verified regression: closed frontier vs open sub-35B, ordinary least squares with 95% confidence bands, frontier fit and all-points fit side by side

Live, interactive version: SWE-bench Verified over time on the dashboard.

The honest caveat, stated up front: if you instead fit every release rather than the frontier (including the dozens of small research checkpoints that exist only to post a number), the open slope still leads (29 vs 22 points/year) but the gap no longer clears statistical significance (p = 0.15). That is the conservative floor, and it exists because the small-model cloud is noisy. It does not reverse the direction; it just widens the error bars. The frontier comparison is the one that matters for "what is the best model I can run," and there the acceleration is unambiguous.

2. The gap has measurably shrunk

Concretely, tracking the best-available closed model minus the best-available open sub-35B model on SWE-bench Verified (watch it move on the live SWE-bench chart):

Date Best closed Best open ≤35B Gap
Jan 2025 49 21 28
Jul 2025 73 54 19
Jan 2026 81 68 13
Jul 2026 96 77 18

The gap fell from 28 points to 13 through early 2026, then widened back to 18, because the closed frontier posted a late spurt to 95+. But that spurt is exactly where you should be suspicious, and section 3 explains why: on the raw, public SWE-bench Verified numbers, the top of the closed leaderboard is inflated by contamination and scaffold tuning. On tasks the models have never seen, the recent closed "spurt" mostly disappears.

3. On the benchmark that can't be gamed, the gap is nearly gone

SWE-bench Verified has been public long enough that its problems leak into training data, and closed labs tune their agent scaffolds hard against it. SWE-rebench fixes this: it pulls fresh GitHub issues every month, so no model has trained on them. It is the closest thing to an honest measurement in this space, and it tells a much sharper version of the story.

  • GLM-5.1 and GPT-5.5 tie for the top overall score at 62.7.
  • Qwen3.5-27B, small enough for one workstation GPU, scores 58.9. That is 3.8 points off the global best.
  • Claude Opus 4.8 scores 56.5, below the 27B open model, despite its 88.6 on the contaminated benchmark.

SWE-bench Verified vs decontaminated SWE-rebench: closed models lose far more of their score than open models when the benchmark can't be gamed

Live, interactive version: SWE-rebench decontaminated leaderboard on the dashboard.

That 32-point drop for Opus 4.8 between the public benchmark and the decontaminated one is the real story hiding inside the "closed frontier is pulling ahead" narrative. A large share of the closed lead reported in vendor announcements is an artifact of the benchmark being public. Strip that away and a 27-billion-parameter open model is already within four points of the best model on Earth.

4. Why this keeps happening: the distillation flywheel

The open sub-35B trajectory is not luck. Every closed or large-open advance gets reproduced at small scale within months, on a public recipe:

SWE-Gym-32B (20.6, Dec 2024) -> R2E-Gym-32B (34.4) -> Skywork-SWE-32B (38.0) -> DeepSWE-Preview-32B (42.2) -> SWE-HERO-32B (62.2) -> Devstral Small 2 24B (68.0) -> Qwen3.6-27B (77.2, Apr 2026).

The techniques are all open: R1-style reinforcement learning, large-teacher-to-small-student distillation (SWE-HERO trains a 480B teacher down into 7B/14B/32B students), and reward-model-guided test-time search (SWE-TRACE takes a 4B base model to 40.7 through inference-time scaling alone). Raw code generation is close to solved at small scale: on LiveCodeBench (live dashboard chart) the open sub-35B frontier is at 77, against a ceiling around 91-93. Closed labs advance; open reproduces; the lag is measured in months and it is not lengthening.

5. About the "crossover date": we're not going to pretend to know it

The obvious next move is to extend the two lines and announce the month they cross. We ran that calculation honestly, and the honest answer is that the exact date is not knowable from this data. Two rising lines with similar slopes have an intersection point that is wildly sensitive to tiny changes in either slope, a textbook unstable estimate. Bootstrapping it 20,000 times gives a median around 2027 but a 95% interval that runs from mid-2026 to well past 2028 depending on which reasonable modeling choice you make.

Bootstrap distribution of crossover-date estimates across three methods, showing wide and method-dependent spread

The dashed extrapolations these estimates come from are drawn live on the SWE-bench Verified chart.

So we don't headline a date, and you should distrust anyone who does. The robust, decision-relevant facts are the ones that don't depend on extrapolating a fragile intersection: open is improving faster, the gap is shrinking, and on decontaminated tasks it is already small. Those hold across every method we tried. The calendar date does not, and it doesn't need to for the conclusion to matter.

6. The one gap that is not closing: tool calling

Here is the honest weakness in the open case, and it is a real one. BFCL v4 measures whether a model reliably selects and formats tool calls, the skill an agent leans on every single turn. This gap is the widest on the board and it is not narrowing:

  • Anthropic: 77.5
  • Google: 72.5
  • Open sub-35B: 51.4

BFCL v4 tool-calling accuracy: Anthropic 77.5, Google 72.5, open sub-35B 51.4, a 26-point gap

Live, interactive version: BFCL v4 tool-calling chart on the dashboard.

A 26-point gap matters more than it looks, because agentic work compounds it. Success over a long session is roughly (per-turn coding correctness) x (per-turn tool-call reliability), repeated hundreds of times. A 5% per-call failure rate alone compounds to near-certain episode failure by turn 60. This is the real closed-lab moat: not model scale, but the billions of proprietary agent trajectories captured from shipped products like Claude Code and Codex, a dataset with no open equivalent. There is early evidence it is narrowing (LFM2.5-8B-A1B hits 64.8 on BFCL v3, Nemotron-3-Nano-4B hits 61.1), but the sample is far too small to forecast. If open closes this gap, the coding-agent race is effectively over. If it doesn't, all the SWE-bench progress in the world stays trapped behind unreliable tool use.

7. What to actually watch

Forget the crossover date. Watch these three, in order of how much they'd move the argument:

  1. Sub-35B BFCL v4 above 70 by Q1 2027. The single most important number in this whole analysis. It is currently the only thing standing between "open codes as well as closed" and "open can be trusted with an agentic loop."
  2. Sub-35B within 2 points of the SWE-rebench #1 by December 2026. Currently within 4. This is the honest capability gap, and it is nearly closed.
  3. Sub-35B above 85 on SWE-bench Verified by December 2026. The flashy public number. Least meaningful of the three, but the one that will make headlines.

The bottom line

Open-weight models you can run on your own hardware are improving at roughly 1.5 times the rate of the closed frontier, they are already within a few points of the best model in the world on tasks that can't be gamed, and the one thing holding them back, tool-calling reliability, is a data problem, not a scale problem. You do not need to know the exact month the lines cross to see where this is going.

The call to action: build an open tool-call trace commons

Everything above points at one lever. The open ecosystem has already proven it can reproduce every closed advance within months (section 4), with a single exception: tool-call reliability, because that is the one capability whose training data is not public. Closed labs did not reach 77 on BFCL by having bigger models. They reached it by training on billions of real agent trajectories harvested from their own products. That is the whole moat. It is made of data, and data is the one thing a distributed community can gather at a scale no single company can match.

So here is the project worth doing, stated concretely.

Build an open agent harness that collects anonymized tool-call traces. The scaffolds already exist: OpenHands, aider, Cline, Continue, and others run millions of agent sessions a month. What is missing is an opt-in telemetry layer that captures the one artifact that actually matters for training tool use.

  1. The trajectory, not the payload. For each session, log the sequence of tool calls: which tool was invoked, the shape of its arguments, the order, the points where the model chose to call or not call, the tool's return status, and how the model reacted to errors. You do not need the user's source code or data to learn this. Redact file contents, secrets, paths, and identifiers at the source and keep the structural skeleton. A trace teaches a model when and how to call a tool even with every payload stripped down to its type.
  2. The success label. The hardest signal to obtain and the most valuable one: did the whole episode actually work? Closed labs get this for free from their products. An open harness can capture it from test-suite results, a passing build, or explicit user acceptance. That label is the reward signal that turns a pile of traces into a training set instead of a log file.
  3. Pool it in the open. Aggregate consented, anonymized traces into a public, permissively licensed, versioned dataset on HuggingFace, and let it grow continuously the way the leaderboards do. Transparency about exactly what is collected is what earns the opt-ins.
  4. Train on it in the open. Supervised fine-tuning on the successful trajectories, then reward modeling and reinforcement learning on the success labels. This is precisely the recipe the closed labs run on their private data. The only missing ingredient has ever been the data itself.

The economics favor open here in a way they never did for pretraining. A frontier pretraining run is a capital problem only a handful of companies can solve. An agent-trace commons is a coordination problem, and coordinating a large community of developers is exactly what open source is good at. There are already more people running open coding agents than any one lab has product users; their combined, consented traces would dwarf any single proprietary corpus.

This is the highest-leverage contribution anyone in this ecosystem can make right now. Not another 32B checkpoint that nudges SWE-bench up two points, because the raw coding ability is nearly solved already (sections 3 and 5). The one thing standing between open models that can write correct code and open models you can trust to run an autonomous agentic loop is tool-call reliability, and that is a data problem waiting for someone to organize the data. Build the harness, collect the traces, publish the dataset, and the last moat drains itself.

Sources & further reading

Regression, gap-closure statistics, and figures generated 2026-07-07 from the seed dataset behind the dashboard above (N=31 closed, N=20 open-<=35B SWE-bench Verified points, Dec 2024-Jul 2026). Frontier slopes via ordinary least squares on each group's running-best subset; slope-difference significance via a two-sample z-test on the OLS standard errors; crossover interval via a 20,000-resample residual bootstrap. numpy / scipy / matplotlib, no external statistics dependencies.