The Scariest Part of AI Is How Confidently It's Wrong — So I Made Four AIs Grade Each Other
I'm not a developer. I use AI to research the things I write, and at some point it started to scare me: AI states wrong answers with total confidence. So instead of trusting one AI, I sat four of them at one screen and had them grade each other's answers. My AI assistant did all the actual building. Here's what worked — and the bug that ate two days, told honestly.
1. The problem — every AI says something different
When I write from research, my habit was this: throw the same question at several AIs, then compare the answers by eye. Switch tabs, copy-paste, hunt for where they disagree.
The whole point is to catch "which AI got it wrong," but the method was primitive. Bouncing between four tabs, I'd miss things, and above all it took forever. Then it hit me: why not make the AIs do that comparison themselves?
2. The fix — I sat four AIs in one meeting room
Here's what I asked my assistant for: "When I enter one question, ask four AIs at the same time. Show each answer as a card the moment it arrives, and at the end summarize what they agreed on and where they split."
It built it like this. Each AI doesn't just reply in prose — it answers in a fixed format: a verdict (true / partly true / refuted / can't verify), a confidence level, the key evidence, corrections, and sources. Cards pop in one by one as answers land, and a summary card appears once they're all in. Results I like save to my notes with one button, ready to cite while writing.
The four are deliberately different in character. One is strong at searching the web and citing sources, two are strong at knowledge and reasoning but trained up to different dates, and the last is another kind of large model. I learned one thing here: four different AIs cover each other's blind spots far better than four identical ones.
3. The core design — don't count a silent AI as a "yes"
My assistant insisted on one principle. If an AI breaks the format or fails to answer properly, you must not count that as a "neutral yes." That inflates the consensus into a fake one. An AI that couldn't answer should drop out as an abstention, not a vote.
And one more. If the four split roughly down the middle, don't force the final verdict into one side — mark it honestly as "divided," and slash the confidence number. "This is contested" beats "an AI confidently wrong" a hundred times over. Honestly, those two rules are basically the whole tool.
4. The highlight — the AIs disagreed, and that was the right answer
As a test I fed it this claim: "Tesla sold the most electric vehicles in the world in 2024."
The result was fascinating. The web-searching AI called it "true" and piled on sources. The reasoning-heavy AI called it "only partly true," pointing to the well-known dispute where the number-one company changes depending on the counting method (annual vs. quarterly, pure EVs only or not). So the final verdict came out "divided."
But that's actually the accurate state of things. The 2024 EV sales crown is a genuinely contested topic that flips with the counting method. Had I asked only one AI, it would have confidently picked a side and I'd have believed it. The mere fact that the four split was itself a signal: "be careful here."
Conversely, a solid fact like "the 2024 Nobel Prize in Physics went to Hopfield and Hinton" got a unanimous "true" from all four. Certain things stay certain, fuzzy things stay fuzzy — exactly what I wanted.
5. The bug that ate two days — "but it works in my terminal"
The feature worked, but one of the four AIs kept not answering and getting marked as an abstention. It would burn the full 240 seconds and return an empty reply. The strange part: when I ran the exact same thing by hand from a command line, it answered in 4 seconds.
This is the kind of bug that makes you lose your mind. "It works when I do it, but not when the program does it." My assistant dug through the logs and found a single clue: the AI tool was frozen, "waiting for additional input."
The cause, in my own words: that tool has a habit of leaving its input window open, as if asking "anything else to add?" When I run it myself, it's a window meant for a person, so the system automatically goes "nope, done" and moves on. But when a program runs it instead, nobody sends that "done" signal, so the tool waits forever.
The fix was almost insultingly simple: one line — right after the program calls that tool, shut the door with "input's done." What's funnier: this same part was also used in the backup path of another AI assistant I use daily. So for months there was a lurking bug where "that AI's fallback occasionally died silently," and this project happened to drag it into the light.
The lesson: "but it works in my terminal" usually means the environment is different, and that difference hides where you can't see it. Even as a non-developer I learned this for good — between "works" and "doesn't," there's always one condition I couldn't see.
6. The numbers
- Measured response times: web-search type ~9s, reasoning type ~16s, the problem AI 16.8s (after the fix — 240s hang before), the last one ~6s
- Bug fixed: one months-old lurking bug, fixed in exactly one line
- Safety net: 21 new automated tests (all with fake responses, verified without real AI calls)
- Timeline: from planning to live-server verification in one day
7. What's left
Now, when I write from research, I don't take one AI's answer at face value. I enter the question, watch what the four say, and the moment they split, I get suspicious right there. It became a useful tool not because the AI is smart, but because it doesn't hide when they disagree.
I still can't read code. But the one rule — "don't trust a single AI; make several of them grade each other" — feels like it'll stick with me long after, developer or not.
