anthill: An Agent Factory That Publishes Its Own Error Rate

anthill is a code reviewer that never stops. It hands every changed version of every tracked file to a Qwen3-8B model running on a Mac mini, and periodically has Claude Opus grade the results. In its first 714 hours it completed 8,826 review runs across 2480 distinct versions of 1291 files, and pushed 24.88 million prompt tokens through the local model at no marginal cost.

Then the paid audit looked at 715 of its findings. It upheld 12 and rejected 703. That is the case study. Local inference solves cost, privacy, and availability; it solves nothing about quality. Running more of it would not have helped, because volume was never the constraint.

What makes the design worth keeping is that the model is allowed to be wrong cheaply, and what the numbers say next is that it is not yet cheap enough. 715 findings reached the layer that bills, and 703 of them should have been stopped before it.

The full argument, with the numbers in context: "What a Local Model Is Actually Good For"

The reviewer is one agent of nine. The part of this system that took the longest to make safe is the part that writes the others: a person types what they want done, a cloud model plans it, writes it, and the result lands in this repository.

The whole safety argument is one property, and it is a test rather than a promise. A generated agent is merged but never scheduled. Merging is reversible, so a bad agent is one git revert away. Scheduling is not, so the machine is never handed it. Two tests hold that line, named so nobody has to read an implementation to know what they guarantee: test_scheduled_is_always_false_across_every_outcome and test_the_module_cannot_schedule_anything.

Planning and building are separate programs. The planner produces a plan a human approves, holds no authority, and writes no files. The step between them names itself the trust boundary, and the distinction it draws is the one most agentic systems miss: running a model with actions disabled makes it safe as an actor, and says nothing about what it returns. So every build-shaped field that comes back, the steps, the reads, the permissions, the capability, is machine-verified or refused. Model output is untrusted input.

The acceptance gate returns the failure text rather than a boolean, because that text is what a bounded retry feeds back to the model that wrote the agent. A boolean tells it that something is wrong; the text tells it what.

The three that exist each answer a question about this machine: which files fill Downloads and which of them are copies, what starts without being asked, and what sits uncommitted or unpushed across every repository here. They live in their own namespace rather than beside the hand-written modules, so generated code is separated from authored code by construction rather than by convention.

Two Macs, one Ethernet cable. The mini has an M4 and 16 GB; the MacBook has an M5 and 24 GB. On the day this was set up the mini sat at 83% memory free with 1.2 GB of swap, and the MacBook, the larger machine, was at 34% free with 18.3 GB of its 19.5 GB swap in use. The smaller computer had room to think, so it became the inference host.

The cable runs at 1000baseT, roughly 100 MB/s measured. Unified memory moves about a thousand times faster, so pooling RAM across the two machines is not possible and was never the plan. Only text crosses. The model lives where the memory is, the work lives where the code is, and the arrangement removes a distributed queue and a SQLite-over-NFS corruption path in one move. mlx_lm.server holds the model at roughly 16 tokens per second; a real 14.6 KB diff takes 6.2 seconds.

The cost curve is what changes the design. A hosted model charges per unit of work, so every review is a decision. A local model charges for hardware once, so additional reviews are free and time is the only limit. Measured capacity here is about 14,000 reviews a day against a real need of roughly 250. There is no reason to ever throttle it, so the reviewer reads every version of every file rather than only the ones a pull request draws attention to.

Findings pass through 5 filters before anything bills. The order is deliberate: free and deterministic first, expensive and stochastic last. Inverting it means paying frontier prices to discover that a finding quoted code which was never in the file.

The first filter uses no model at all, and it is the one carrying the economics: 572 findings quoted code that was not in the file and died in a string search, for nothing. The middle filters are the measured weak point. They are the next thing to fix, and that is a more useful thing to publish than the token count.

Giving software its own computer forces the safety question on day one. These are constants and refusals in the source rather than intentions in a document.

The first coordination mechanism was a heartbeat table: one row per agent, overwritten. It answers the wrong question. The dangerous state is not that a process holds a lock, it is that a process held one and then died, and an overwritten row keeps no history and cannot say what anyone is holding.

It became a small bus with four primitives: what is true, what is held, an append-only log to tail, and what the human has permitted. Everything expires, so the mechanism that recovers from a dead agent is the same one that grants a live agent permission. The model server's own file lock is enforced by the kernel and visible to no other process, which is exactly how the memory steward would otherwise have frozen the reviewer mid-lock and deadlocked both.

Presence is scoped per machine for the same class of reason. A single global "someone is at the keyboard" flag made the reviewer ease its batch from 25 to 3 because a human was typing on the MacBook, while the work it was easing ran on the mini, where nobody was sitting. An unmeasured machine now returns unknown, and unknown means callers change nothing.

The question this started with was whether a small model on cheap hardware could stand in for a large model on someone else's. It cannot, and 12 of 715 is the number that settles it.

The framing was the mistake. A local model is not a discounted frontier model; it is a different instrument, one that produces candidate volume at no marginal cost on hardware you control, with no quota and no queue. 3174 findings that are mostly wrong is noise. It becomes useful only once something can discard it for free, which makes the 572 findings that died in a string search closer to the real result than the five million tokens.

The 12-of-715 number is a measurement, not a verdict. The move after reading it was to find where the loss happened rather than to argue with it, and the answer was already sitting in the vote table. The three lenses meant to challenge every finding had cast 90 votes and refuted 0 of them. A verification stage with a 0% refute rate is not verifying anything. It is a rubber stamp with a latency cost, and it is how 703 findings walked into the layer that bills.

Those lenses were replaced on 2026-08-01 with three that have to cite an artifact before they are allowed to agree. Across 3,367 votes the new set refutes 1213, and the spread between them is the part worth having: one refutes at 57.4%, the next at 36.9%, the third at 4.1%. Three lenses that disagree with each other are asking three different questions. Three that never refuse anything were asking one question and hearing one answer.

What has not moved is the number that matters, and the update is worse than the previous version of this paragraph rather than better. The 42 audits since the rebuild have reviewed 653 findings and upheld 9 of them. That is 1.4%, against 3 of 62 before it, which was 4.8%. The refute rate upstream went from nothing at all to 1,213 of 3,367 votes, and end-to-end precision fell anyway. Presenting the improved refute rate as a win would repeat the exact error this case study is about, which is mistaking an intermediate metric for the outcome.

It is still the tractable kind of problem, and that is the reason to keep turning the crank rather than start over. The failure lives in prompts, thresholds, and ordering rather than in model weights, so an iteration costs an afternoon instead of a training run. Every rejection is stored with the auditor's reasoning, which means there are 703 labelled negatives sitting in the same database as the findings they describe. The audit layer already writes rules back into the reviewer, and it has written 203 of them so far. The loop that has to close is built and turning. It has not produced a measured gain yet, and until it does, that is what this page will say.

The cross-model judge carries its own warning label. It refutes 355 of the 393 findings it has seen, and a rate that high is watched rather than celebrated: near zero would mean it had started rubber-stamping like the lenses it replaced, and near 100% would mean the same one-directional bias pointing the other way. The question being tracked is not how often it disagrees, but whether it disagrees for reasons that survive being checked.

Adopting a protocol is a decision that can be wrong in both directions, so each was checked against this system's actual threat model and against the current spec rather than against its landing page. 5 decisions came out of it, and only one of them is an adoption. Each carries the trigger that would reopen it, because a decision without a revisit condition is just a preference.

One spec reading settled the largest of these. The 2026-07-28 MCP revision deprecates Sampling and Roots, which forecloses "the second machine joins the bus over MCP" by specification rather than by preference, and the documented replacement, talking to the model API directly, is already what this system does. Reading the changelog was cheaper than discovering that in an implementation.