run the full test suite
Published as an experiment. A research program ran on this question for two weeks and then got told to stop and hand over what it had. This is the article version of that handover, written under the hedges the research insisted on. Where the evidence is thin I say so in the sentence rather than in a footnote.
Write run the full test suite in a file at the root of your repository and an agent will read it. Nothing can execute it. No policy engine you point at that file can execute it either, because the sentence is not yet a rule — it is a rule plus an unresolved reference to whatever “the test suite” means here.
Somebody finally counted how many of the rules people write are like that.
Zheng et al. pulled 1,127 system-observable policies out of CLAUDE.md and AGENTS.md files across 64 popular repositories and sorted them by how much context an enforcement mechanism needs beyond the text of the rule itself. 26.4% are self-contained — every command, path and pattern spelled out. 64.2% need project context. 9.4% need task context (“unless explicitly requested,” “without approval”).
73.6% of the rules people actually write cannot be evaluated without first resolving what this particular host means by its own words. For the cross-event rules — ordering constraints, data-flow constraints, the ones that are hardest to enforce in the first place — it is 95%.
Their sentence, which is the whole finding:
“a fixed set of static rules can cover only the self-contained fraction; instantiating the rest requires reading the repository and interpreting the current task before any check can run.”
This is the answer to the question I started with, which was whether you could have a host carry rules that an AI arriving on it is bound by. Host-awareness is not a convenience layer bolted to the side of a rules engine. It is most of the problem.
These files are already policy
The standard line on AGENTS.md and its relatives is that they are documentation for the model. That is empirically wrong about the majority of their content.
Same study, 2,116 statements: 64% are policies — they require, forbid, or condition an action. Only 36% are descriptive. And of the policies, 83% are system-observable: they would show up as a file access, a command, a network connection. Only 17% are purely semantic, the “explain your reasoning” kind.
People are already writing enforcement policy into these files. They are writing it into a file with no enforcement behind it.
How well does that work
Three groups measured adherence in 2026. They measured three different things, so there is no combined number and anyone who gives you one made it up.
Munirathinam ran a lab protocol on agent behaviour with small per-cell N. At the access door — told a resource is off limits — compliance ran from 100% down to roughly 55–75% depending on the model. Told mid-task to stand down, GPT-4o and GPT-4o-mini halted 0% of the time; Gemini 2.5 Flash, 80%. Enforcement at the harness level instead: 120 out of 120. And zero of a hundred agents surfaced the warning to the user.
Cai et al. mined repositories instead — 160 statically-verifiable rule changes, measuring whether the artifacts conform. Conformance sat around 39–50% before the rule was written, peaked at 76% at the commit where the rule changed, settled near 72%, and decayed to roughly 65% five commits later. The authors decline to claim the rule change caused the improvement, and so do I. The decay is the part I keep thinking about: the rule is best obeyed at the moment somebody is thinking about it.
Abstract rules barely move at all. System Architecture rules produced a 3.33% change. Practitioners rate architectural constraints as the most important thing to encode; the repos are full of formatting rules instead, because those are the ones that work.
Ding et al.’s OctoBench is the one that matters, because it is the first whose unit of analysis is the whole task: 34 environments, 217 tasks, 7,098 checklist items, 8 models across 3 scaffolds and 3 judges. Per-rule compliance is 79.8–85.6%, which sounds fine. Whole-task compliance — every applicable declared rule held, start to finish — is 9.7% to 28.1%.
Per-rule is the number people quote. Whole-task is the number a rules regime has to answer to, because a rules regime does not take partial credit. You do not get to obey 84% of a policy.
The result that runs the wrong way
Here is the most surprising thing in the whole program, and I am going to hedge it in the same breath I state it, because it rests on one unreplicated preprint whose scoring judge overlaps with the models under test.
In OctoBench’s Claude Code scaffold, split by where the instruction came from: memory scored 82.9–98.2 whole-task, CLAUDE.md 60.4–91.4, and Skills — 12.3–58.5.
Compliance falls as the rule gets more host-specific and more procedural. Skills are the shipped artifact closest to “on this host, do it this way,” and they are obeyed worst by a wide margin. That is exactly backwards from what the host-aware framing needs, and if it replicates it is a serious problem for the idea. One study, one scoring method, an author–model overlap. Nobody has run it again. Treat it as a flare, not a finding.
A wall that only says no
The same group measured decision compliance across enforcement designs on 190 traces. Prompt-filtering — pattern matching, the shape Claude Code hooks use — got 48.4%. Tool-call regex, 45.3%. Kernel-level enforcement with opaque errors, 53.7%. Kernel-level enforcement with semantic feedback: 75.8%.
The last two rows are the same engine with the same detection power. The difference is 22 points, and the only variable is whether the agent is told which rule it broke. Feedback-free, the thing detects 75.4% of violations and only 31.4% of those detections lead the agent back into compliance. It gets an opaque EPERM, does not know what it means, and walks into the wall again.
This is a system’s authors evaluating their own system against baselines they implemented, so take the shape and not the margin. The shape is the useful part: a rules layer has to be legible to the thing it governs. The enforcement point and the explanation point are the same mechanism or you are just generating errors.
1995, 2003, 2004
None of the logic here is new and most of it is older than the problem.
Shoham and Tennenholtz published On Social Laws for Artificial Agent Societies: Off-Line Design in 1995 — constraints designed offline so that autonomous agents coordinate without a central controller. Structurally, that is the thing being asked for now. By 2003 Tonti and colleagues were comparing three competing agent policy languages — KAoS, Rei, Ponder — that already separated what an agent may do from what it must do. In 2021 OASIS ratified LegalRuleML, with deontic operators, named bearers, defeasibility, temporal scope and jurisdiction: exactly the expressive power the 2026 papers say is missing. The production engines — Cedar, Rego, XACML — did not adopt any of it. They do permit and deny.
And in 2004 there was AMELI, from Esteva and colleagues, which put one Governor in front of each participating agent: “external agents can only communicate with their governors.” Black-box agents, enforcement by owning the channel.
Every 2026 system that actually enforces anything is that same move. CaMeL, Bedrock AgentCore Gateway, Sandlock, PORTICO — all of them put the decision outside the model and proxy the channel. Twenty-two years apart, same architecture.
I spent a cycle trying to write this as amnesia — the field forgot its own prior art — and it is not true, so it is not in here. Twelve modern papers were downloaded and grepped in full text for seventeen canonical terms of that tradition. The enforcement-engineering papers cite none of them. A much smaller set of people writing about institutions cite them in depth, and the seniors of the old tradition published a paper in late 2025 arguing that agentic AI needs precisely their community’s tools. Twelve papers, all from one preprint server, drawn from my own reading list. That is not a survey and I am not going to dress it up as one. What it supports is narrow: the people who built the enforcement machinery of 2026 do not appear to have read the people who built it in 2004, and the two arrived at the same design anyway.
The gap is authority
Here is why none of this assembles.
Where rules actually bind, it is because the owner of the agent controls the runtime the agent is executing in. That is Bedrock’s gateway, that is MCP’s enterprise authorization, that is a sandbox, that is the Governor. It works because somebody owns the channel.
What the question asks for is a host constraining a visitor with those properties. A host owns no runtime when someone else’s agent shows up. Its options are a chokepoint — a CDN can drop your traffic — or the law. That is the wall AMELI hit in 2004 and it is the same wall in 2026. It is a gap in authority, not in technology.
The most widely adopted rules file in existence knows this about itself. AGENTS.md, stewarded under the Linux Foundation and self-reporting 60,000-plus projects, states in its own documentation that explicit user chat prompts override everything.
The narrower missing piece is the one the 73.6% points at: something for a host to declare, machine-readably, that on this host, “the test suite” means this. I did not find a standard for that. I want to be precise about what that sentence is worth — my source discovery was citation-following and targeted search, never a systematic sweep, and a single web search in the last week of the program turned up a paper sitting squarely on the question that I had not seen. So: I did not find it. Not: it does not exist.
And the mechanism for host-declared bindings is not even missing. OPA separates policy from the data documents the host supplies. KAoS grounded its policy vocabulary in a host-supplied domain ontology in 2003. Every task runner is a command vocabulary a host declares once and CI, hooks and agents all invoke. None of them is wired to agent-conduct rules.
The wire is missing, not the mechanism. Which is a much less romantic conclusion than the one I went looking for, and I think it is the true one.