I built an LLM judge to check my own scorer. It disagreed with me.

I wrote a deterministic index that scores how hard a Terms of Service document works to hide that "Buy now" means "revocable licence". Then I had language models read the same documents and score them independently. I expected a nice validation number. I got a correlation of 0.12.

That's roughly the correlation between two things that have nothing to do with each other.

My first instinct was that I had a bug. I didn't. The disagreement is real. It survives a much bigger model. It turned out to be the most useful thing the whole experiment produced. Here's what happened and what I actually believe now.

The setup

DODI is a keyword scorer. It counts licence words against ownership words, penalises unreadable prose, counts red-flag clauses and weights those 50/25/25 into a 0 to 100 score. Same document in, same score out, no model involved. That determinism is the point: it's what makes a 2015-to-2024 comparison honest.

The judge is the opposite kind of instrument. Each document goes to a language model with a rubric that mirrors DODI's three components. The model returns structured JSON: subscores, an overall deception score, a letter grade and a short rationale. Every response is cached to disk, so the analysis reruns without touching a model. Anyone can check my numbers without paying for tokens.

Three judges. Claude Haiku, small and fast, on all 51 documents. Claude Sonnet, considerably stronger, on the 11 validation documents, ten of which have reviewed human grades from ToS;DR. Then Google's Gemini 3.5 Flash on the same 11, so the check doesn't rest on one vendor. The Claude judges ran locally through the CLI on my normal subscription and Gemini on its free tier, so this cost me nothing but time.

Before looking at agreement I checked the judge was stable at all. Ten documents, three runs each: mean per-document standard deviation of 6.3 points, worst spread 20. Not noise-free, but nowhere near enough noise to explain what came next.

The result

Spearman rank correlations. Three things fall out of that table. I trust them in this order.

One: the disagreement isn't a small-model artifact. This was my main worry. Cheap model, sloppy scores, case closed. So I ran a much stronger model over the validation set. It landed at 0.26: a bit higher, still weak, nowhere near significant. The two models even fail differently. Haiku scores high and narrow, handing out a lot of 68s. Sonnet scores lower and wider, putting most mainstream contracts at "moderate". Different distributions, same verdict: neither of them reads these documents the way DODI counts them.

Two: the readers agree with each other more than any of them agrees with me. Sonnet and Haiku correlate at 0.50. Gemini and Sonnet, from different companies, correlate at 0.72. Against DODI the three manage 0.12, 0.26 and 0.05. There's a rough LLM consensus about these documents. It crosses vendors. My index isn't part of it.

Three: nobody wins against the humans. DODI reaches 0.45 against the human grades and all three judges' scores 0.34. The letter grades do better: 0.56 for Haiku, 0.65 for Gemini. At ten documents and this many comparisons you can't rank these at all.

So this isn't a story about the LLM being right and my scorer being wrong. It's messier and more interesting than that.

Adobe is where you can see it

The clearest disagreement in the corpus is Adobe's 2024 terms. DODI gives it 95.7, the 7th most deceptive of 40 documents. Haiku gives it 44. Sonnet gives it 32. When something is that far apart, one of the instruments is measuring the wrong thing.

Both judges pointed at the same reason. Adobe states plainly that its software is "licensed, not sold". It says you own your content and that it won't train AI on it. And critically, its store prices every plan per month. The button does say "Buy now", but nobody reads a monthly subscription as owning the software.

DODI can't see any of that. It sees a document stuffed with the word "licence" and very few "buy" or "own", so the ratio metric maxes out and the score lands near the top. My index conflates licensing-language density with deception. An honest subscription service that never pretends you're buying anything scores as highly deceptive. That's a false positive. It took a reader to find it, because a counter can't find it by construction.

Which points straight at the fix. It's one I'd already written into the repo's future work without realising it was load-bearing. Ownership deception isn't "how much licence language is there". It's the gap between what the storefront promises and what the contract grants. High only when the button says Buy and the fine print says licence. Under that definition the subscriptions fall away, Adobe halves and the stores that sell titles keep their scores. That's DODI v2. It exists because the judges disagreed with v1.

The judges also caught something I got wrong

Separately (and slightly embarrassingly): one document in my corpus was mislabeled. I had a file called whatsapp_tos.txt. It was Wikipedia's Terms of Use. Forty-three mentions of "Wikimedia", zero mentions of "WhatsApp".

The grade and the service ID attached to it were both Wikipedia's, so the document still matched its grade. Only the name on the file was wrong. My deterministic pipeline had scored that document happily for weeks, because it doesn't read, it counts. A keyword counter has no opinion about whether a document is what you claim it is.

Both judges scored it 6 out of 100 and described a nonprofit agreement for content licensed under copyleft terms. Not a messaging app. The mismatch was obvious the second something actually read the text.

That's the argument for a second instrument in one paragraph. The deterministic scorer is reproducible and blind. The reader is fallible and awake. You want both. You especially want them when they disagree.

What I actually believe now

Three defensible methods for measuring the same construct barely converge. A keyword index, three language models and a panel of human volunteers produce five different rankings of the same ten contracts. None of them agrees convincingly with the humans.

The honest conclusion isn't "use the LLM" or "use the counter". It's that ownership deception isn't one number. Any single automated ToS score, including mine, should be read as one instrument's opinion rather than a measurement. I'd rather publish that than a tidy validation plot.

Where this is weak

• Ten documents. Every human comparison in that table is n=10 and not one of them is statistically significant (p = 0.19, 0.34, 0.34). Only the Haiku-vs-DODI number rests on the full 51.
• Uncontrolled temperature. The judges run through the CLI, so I can't pin sampling. The three-run consistency check measures the judge as deployed, not the judge at temperature zero. Defensible, but it isn't the cleaner experiment.
• One rubric. All three judges read the same prompt. Some of their agreement with each other could be agreement with my rubric.
• The rubric is mine. I wrote both the index and the prompt that grades it. Some shared bias between them is likely, which if anything makes a 0.12 correlation stranger, not safer.

Everything reproduces from the repo. The cached judge responses are committed, so you can rerun the whole agreement analysis without a model or an API key and check whether I read my own numbers correctly. That last part turned out to matter.