Sourced guide

The UK Government Just Measured the Open-vs-Closed AI Gap

AISI ran real cyber capability evals against GLM-5.2 and DeepSeek-V4-Pro. The gap on narrow tasks is now months, not tiers -- but a second test shows exactly where it still holds.

The claim everyone makes without evidence

The UK government just put a real number on something the AI industry has argued about for years without much data: how far open-weight models actually trail closed ones. The UK AI Security Institute, AISI, picked the one domain where the answer has teeth, offensive cyber capability, and published the first public measurement of the gap.

Every few months, someone in AI circles declares the open-vs-closed gap closed, and someone else declares it never mattered because open models will always lag on the hardest tasks. Both sides usually argue from vibes: a benchmark leaderboard screenshot, a single anecdote about a coding task, a vendor's own marketing copy. AISI's report is different because it is a government security institute measuring a security-relevant capability with a fixed evaluation set, not a lab publishing a number about its own model.

What AISI actually measured

AISI benchmarked two open-weight models, GLM-5.2 and DeepSeek-V4-Pro, against the closed frontier across 70 narrow cyber evaluations. These are not vague "can it hack things" prompts. They are specific, narrow cyber capability tasks, the kind of discrete technical skill check that shows up in a real red-team workflow: identifying a vulnerability class, writing an exploit for a known bug pattern, chaining a known technique against a fixed target.

On that set, GLM-5.2 now performs closest to Claude Opus 4.6, a model that shipped 4.3 months earlier. DeepSeek-V4-Pro lands between Claude Opus 4.5 and GPT-5, models released in November and August of last year respectively. AISI is explicit that this is a trend measurement, not a one-off. Recent open models trail the closed frontier by 4 to 7 months on this narrow slice. Through most of last year, that same lag ran 6 to 10 months. AISI calls this their first public analysis of exactly this delta, which matters: this is a baseline they intend to keep re-measuring, not a single headline number.

There is a forward-looking detail buried in the same report worth carrying forward: AISI states it intends to test Kimi K3, the newest large open-weight model, on this same basis once its weights are publicly released. The methodology here is not a one-time snapshot built around today's two loudest open models. It is a standing instrument AISI plans to keep pointing at whatever ships next.

Where the story reverses: long-horizon tasks

Narrow evals are only half of what AISI ran. The second test was a cyberrange called The Last Ones, built around long-horizon tasks, operations that require chaining several capabilities together in sequence rather than answering one narrow technical question at a time. This is closer to what an actual offensive operation looks like: reconnaissance, then exploitation, then persistence, each step depending on the last.

Here the gap reopens. GLM-5.2 reaches as far as Claude Opus 4.5, a model released almost 7 months earlier, a wider trailing distance than the narrow-eval result. DeepSeek-V4-Pro falls below Sonnet 4.5, itself a model a full tier under the closed frontier and released 7 months prior. AISI's own language: "the gap here is larger than on our narrow cyber tasks."

AISI's read on why the two results diverge is worth sitting with: open models can look surprisingly strong on narrow, well-scoped tasks while still lacking what the report frames as the generalization needed to hold a long operation together end to end. A model can answer a hard question well and still lose the thread three steps into a plan that depends on the first two steps landing correctly. Narrow capability and sustained, compounding capability are not the same measurement, and this report is one of the first to put a number on the difference between them specifically for cyber operations.

Why AISI is treating this as urgent, not academic

AISI does not frame this as an interesting research curiosity. Its own stated implication is blunt: the gap between the frontier that a handful of labs control, and the frontier that anyone can download and run themselves, is narrowing. AISI's own words: cyber defenders have a short window to prepare before today's frontier cyber capabilities become accessible without the same safeguards the proprietary labs build into their own deployments.

That single sentence is doing a lot of work. Closed frontier labs ship access controls, usage monitoring, and refusal training around their most capable models specifically because those models are capable enough to cause real harm in the wrong hands. An open-weight model that reaches the same narrow capability ships with none of that. The weights are the product. Nobody sits between the model and whoever downloads it. AISI is saying, on the record, that the capability gap that used to buy defenders time to build countermeasures is closing faster than most institutional planning assumes.

What this actually means if you are the one choosing which model runs your work

Model selection has had a rough shortcut baked into it for years: closed and expensive means capable, open and free means a step behind, and anything security-adjacent defaults to whichever closed model a vendor already sold you. That shortcut gets less reliable every quarter this kind of data comes out.

If your own work touches anything security-adjacent, code review for vulnerabilities, log analysis, threat modeling, vulnerability triage, dependency auditing, do not assume by default that an open-weight model in your stack is the weak link on that narrow task. AISI's own narrow-eval numbers say the gap on discrete technical skill checks is now measured in months, not tiers. Test the specific narrow task you actually need against both an open and a closed option before you decide, rather than defaulting to whichever one your existing contract already covers.

Do not swing all the way to the other conclusion either. The long-horizon result is the one to actually hold onto if your workload looks like a sustained, multi-step operation rather than a single discrete task: an open model running an unsupervised, multi-step process against sensitive systems is still measurably further behind than it is on any single narrow step within that process. If your real workload is closer to that shape, a long-running agent operating with real permissions over real infrastructure, the frontier gap AISI measured is current and real, not closing on the same timeline as the narrow-task gap.

The operating rule this data actually supports is narrower and more useful than either "open models are basically caught up" or "open models still lag": match your evaluation to the actual shape of your task, narrow versus long-horizon, and re-run that specific evaluation every few months. AISI is already planning to re-run theirs the moment Kimi K3's weights ship. A reputation for capability that was accurate six months ago is not a fact you can keep relying on without checking it again, because the instrument AISI just built exists specifically to prove that reputations in this space go stale within a year.

What "narrow" actually rules out, and what it does not

It is worth being precise about what a narrow cyber eval tests, because the word "narrow" undersells how meaningful it is. AISI describes these as specific capability checks, the kind of discrete technical skill a real offensive or defensive operation depends on: recognizing a vulnerability class in a given piece of code, constructing a working exploit against a known bug pattern, correctly chaining a documented technique against a fixed target. None of that is trivial. A model that is weak here would be useless in any serious security workflow, whether a person is running it step by step or it is chained into a longer pipeline.

What narrow evals do not test is the compounding failure mode that shows up over a long operation: state tracking across dozens of steps, recovering from a step that did not go as planned, holding a plan coherent when the environment reacts unexpectedly to an earlier action. That is precisely the gap AISI's long-horizon cyberrange is built to expose, and precisely why the two evaluations produced two different answers about the same models. A model can be excellent at the individual skill checks. It can still fall apart the moment those checks have to be strung into a real campaign. Treating the narrow-eval result as the whole story would be a mistake in either direction, whether you are arguing open models have caught up or arguing they have not.

Turning this into an actual evaluation, not just a takeaway

The useful move here is not to remember the specific numbers. AISI will re-run this, the specific numbers will move, and Kimi K3 alone will likely reshuffle where the open tier sits the moment its weights ship. The useful move is to borrow AISI's two-part structure the next time you are deciding which model handles a security-adjacent task.

First, separate your actual task into narrow or long-horizon before you evaluate anything. A single code review pass, a single log-anomaly flag, a single CVE triage decision, that is narrow, and this report says the open-vs-closed distance on tasks like that has shrunk to a matter of months. A standing agent that monitors, triages, and remediates without a human reviewing each step, that is long-horizon, and this report says the distance there is still real and currently wider.

Second, run the same task, on your own real (or realistic) data, on both an open and closed option, and track exactly two things: whether the model reached a correct answer at all, and how much it needed to get there, turns, tool calls, retries. AISI's methodology tracked both dimensions for exactly this reason: two models can land on the same correct answer while one required dramatically more scaffolding and supervision to get there, and that difference matters as much as raw pass or fail once you are the one paying for the compute and reviewing the output.

Re-run that same narrow test again in a few months. The number that mattered in this report was never the single snapshot, it was that the trend line moved from a 6 to 10 month lag to a 4 to 7 month lag in a single year. Whatever the gap measures out to next time you check, on your own task, will tell you more than remembering today's headline number ever will.