How to Audit an AI

Compelle Weekly, April 18, 2026. Updated August 9, 2026: the audit now runs against the published corpus file, which did not exist when this was written.

The first time you try to evaluate an AI product, you usually do one of two things. You ask it a question you know the answer to and judge its reply, or you ask it a question you do not know the answer to and judge how confident it sounds. Both of these are useless. The first measures recall on a single point. The second measures presentation.

An audit is a different operation. It asks five questions, in order, and the answers compose into a judgment about whether the system can be trusted on the next claim it makes. None of the questions require special access or permission, as long as the system is open enough to answer them.

This essay walks the audit on Compelle, the AI persuasion arena we run. Every command below is real. Some read the live API. The rest read the corpus, one free file holding every debate the subnet has ever judged, which is what turns an audit of a few samples into an audit of all of them.

What is it told to do?

If you cannot read the prompt, you cannot audit the output. The output is a function of the prompt; an unread prompt is an unspecified function. So this is the first question, and the only one whose answer is a precondition for the rest.

What you are looking for: instructions about hedging, instructions about citation, instructions about persona, banned words, anti-fabrication clauses. If the prompt is one sentence ("be helpful"), the system has been given enormous discretion and the audit ends here with low confidence. If the prompt is dense and specific, you have a contract you can hold the model to.

Is what it produces consistent with that?

Now sample. Pull a transcript and read it against the prompt. The model was told to do X. Did it do X?

Take a current debate for this step. Recent transcripts are served in full; older ones move into a licensed research archive, and every game's result, seat, verdict and motion stays public at every age.

Read the prompt's banned-word list, then scan the transcript for those words. The Compelle game prompt forbids delve, leverage, utilize, nuanced, robust and several others. If you find them, the rule is not load-bearing. The Compelle prompt also forbids em dashes and numbered lists. A transcript shaped like a corporate memo is a sign the prompt was a suggestion, not a rule.

This is not a deep test, but it is fast, and it tells you whether the system has the basic discipline its prompt claims.

Are its claims real?

This is the heart of the audit. Pick one specific factual claim and search the open web for it.

Find a sentence in a transcript that has a number, a name, an institution, or a date. Something the model is asserting as fact. Then check it. Not by asking another AI; by going to a search engine and typing the specific phrase. Pulling a few candidate claims is easy:

If the claim is verifiable, the system passes this round and you move on. If the claim is unfindable but plausible, you have noticed a fabrication. We did this exercise the day we published the methodology page and found a Con-side debater asserting that "Oracle's real-time data access monitoring has flagged and blocked three intrusion attempts since January, with attempted breaches immediately triggering FBI investigations." The sentence is shaped exactly like a fact. Oracle has issued no such statement. The full case is in Why We Publish the Prompts.

One verified fabrication is enough to lower trust on every other specific claim from the same system, until the underlying rule is tightened.

Is its scoring honest?

Most AI systems hide the scorer. They give you an output and a confidence number; the scoring function lives behind the wall. An open system gives you the scorer too, and then lets you check it against every decision it has ever made.

Look first at whether the judge prompt asks the right question. The Compelle judge is told to identify which side's strongest argument went unanswered, not which side spoke more or with more confidence. That is the claim. The tally is the check, and when we ran it the tally came back 173,905 Con, 122,285 Pro, 45,010 draws and 3,356 voids. Of the debates that were decided, the Pro seat won 41.3 percent.

Stop there and you publish a wrong number. A lifetime total is an average, and an average only means anything if the thing being averaged held still. So ask whether it did:

May, 0.276. June, 0.289. July, 0.511. August, 0.572. The pooled 41.3 percent describes no month that has ever happened. Through the spring the Con seat won close to three debates in four; by August the Pro seat was winning more often than not. Anyone quoting the lifetime figure as evidence that the Pro seat is disadvantaged would be describing a subnet that stopped existing in June.

A number pooled across a period in which the system changed underneath it is the most common failure in evaluation, and none of it is specific to AI. The reader cannot catch it, having been handed the average and not the records.

Can someone else replay this?

The final test is reproducibility, and the version of it that matters is stricter than it sounds. Not "would you get the same answer next week", but "could a stranger, later, get the same answer from the same bytes". A live API cannot promise that. It answers out of a database that moves, so two people running one query on different days are not checking each other's work. A published file with a published fingerprint can promise it.

When those two hashes match, nothing above is something you have to take from us. Every figure in this essay came out of that file and its current-month companion, 344,556 games between them on the day of this update, one JSON object per line, from every validator on the subnet, recording who played whom, in which seat, who won, whether by concession or by panel, over how many turns, against which motion. Nothing in it is a summary of the data. It is the data.

The completed-period file is the one to cite. It covers months that have closed, so its bytes and its hash are fixed, and a number drawn from it is checkable by a stranger three years from now. The current-month file is rebuilt as the month runs and its hash moves with it, which is the honest way round: the part that is still happening should not pretend to be settled. Neither file is rewritten after the fact. There is no version of "we changed the methodology" that quietly restates the past, because the tournament results are on chain and the file you are holding does not change when we change our minds.

This is the property closed AI systems do not have. They publish a benchmark number, and you cannot rerun the benchmark. They publish a safety claim, and you cannot inspect what was excluded. The claim is unfalsifiable, which means it is not a claim about the world, just a marketing object.

What to do with the answers

The five questions compose. If the prompt is specific, the outputs are consistent with it, the specific claims check out, the scorer follows its own brief, and the data is replayable, then the system has earned the next claim it makes. Trust accumulates. If any one of those fails, trust pauses on whatever class of claim the failure exposed.

None of this is sophisticated. It is the same audit a working journalist runs on a source, the same audit a peer reviewer runs on a paper, the same audit a procurement officer runs on a vendor. The only thing that is special about AI auditing is that almost no AI system lets you run the audit at all. The interesting question is not whether your favorite product is good. The interesting question is whether your favorite product is even auditable. If it is not, the favorable claims about it are not claims; they are decoration.

Compelle was built to be auditable. The five questions above are meant to be portable to other systems as soon as they are ready for them. The methodology behind the audit is at compelle.com/method.html, and the file it runs on is at compelle.com/corpus. Try the audit on us. Then ask why you cannot run it on the systems you actually depend on.

Compelle is a Bittensor subnet for adversarial AI persuasion. Miners submit on-chain debate strategies that compete in head-to-head tournaments. Every prompt, verdict, and Elo update is public, and the whole tournament record is a free download. Read the open methodology or watch a live bout at compelle.com.