The Thesis, and the Question Behind the Question
In four published product runs, the loop shipped when the judge was independent of the author, measured gates could not be waived by a score, and ties counted as no progress. The write-up never tests them as necessary conditions (Sean Carey’s design-loop write-up).
The surface question is how agents learn from feedback. The question behind it is what makes feedback trustworthy enough to learn from: who judges the judge, when does the loop stop, and what can a score never buy? The source anchors quality in captured reality — the coordinator researches leading products and records real screens, and the product’s own baseline is captured the same way — so a judge saying “9” has to mean “comparable to these”, not “nice”. The practice guide adds a complementary constraint: “If you ask for general feedback, you usually get general taste language” (Agentic Design School, Design Critique Loops With Agents).
Two scope notes. On differentiation: a 2026-08-16 post, “can design quality be measured”, and a 2026-11-08 post on a kanban review pipeline covered measurement and pipeline; this post covers the stop rule, judge independence, and gates-versus-score. On honesty: this blog did not run the tools or the loop hands-on; everything below rests on the published write-up and the cited papers. Where this blog’s own arena uses a mean gate while the source’s rule is a panel minimum, the difference is reported here, not adjudicated.
Four Computable Rules
Each condition is enforceable in code: the coordinator computes the weighted score, the panel result is the lowest judge’s score, measured checks are gates no score can waive, and two consecutive non-improving rounds stop the run. The skill (design-loop) and the write-up (Sean Carey) supply the base rules; items marked (this blog’s addition) are ours, derived from the author’s warnings.
R1 — Panel-minimum scoring
PASS: the recorded panel score equals the minimum across all judges’ scores on the same frozen candidate (the frozen-candidate requirement is this blog’s addition). FAIL: any mean, median, or max aggregation (this blog’s addition), or a changed candidate inheriting old scores.
R2 — Gate veto
PASS: all measured checks green — contrast at 4.5:1 or better, no horizontal overflow, keyboard focus on every Tab stop, reduced-motion respected, repository tests passing. FAIL: any check red, regardless of rubric score, because “A high score can’t waive a failed check.”
R3 — Stagnation stop, ties count as no progress
PASS (continue): the primary judge’s score (phase 1) or the panel minimum (phase 2) strictly increases from one round to the next. FAIL (stop): two consecutive rounds without strict improvement end the run. A tie increments the stagnation counter and never resets progress (this blog’s wording of the author’s “ties don’t count”). Worked example, phase 1 (primary judge only): Tanks. Gemini scored 9.0 in rounds 1–3, but round 1 failed CSS and JavaScript size limits, round 2 failed browser tests, and round 3’s action log showed no coverage of station tabs or keyboard chart access. Rounds 2 and 3 were consecutive ties at 9.0 — “two ties end the run” — and the run stopped.
R4 — Judge independence
PASS: the primary judge comes from a different model family than the code author; judges are read-only, drive the actual UI, and bring fresh screenshots; judges are blind to each other’s scores; one same-family judge at a higher reasoning level is allowed. FAIL: a primary judge from the author’s family, a judge that reads code or the coordinator’s screenshots instead of the product, or a mid-run swap to a more forgiving judge (this blog’s addition, from the author’s warning).
The contract below is the bridge to the findings registry at src/data/design-findings.json — one frozen candidate, its gates, its panel rule, and its stagnation state, checkable mechanically:
{
"run": "example",
"candidate_hash": "<frozen build>",
"author_family": "<code author's family>",
"judges": [
{"id": "judge-a", "family": "x", "read_only": true, "fresh_screens": true, "score": null}
],
"gates": {"contrast_min": 4.5, "h_overflow": "none", "tab_focus": "complete",
"reduced_motion": true, "repo_tests": "pass"},
"panel_score_rule": "min",
"stagnation": {"tie_counts_as": "no_progress", "stop_after": 2, "current": 0}
}
How to Verify Each Rule
Verification needs artifacts, not vibes: per-round HTML reports, judges’ own screenshots and accessibility dumps, judge action logs, and code-measured gate outputs — the same evidence trail the four-run series produced (Sean Carey’s design-loop write-up).
- R1 → round reports. Every round yields an HTML report; success ends in a PR with independent code review, failure ends in a draft titled GOAL NOT MET. Verification means scores from every judge on the identical candidate, with the coordinator doing the arithmetic so no judge can hand in an overall vibe score. The site rubric weighted task value 15, clarity 15, efficiency 15, control 15, visual craft 20, emotional appeal 10, and brand fit 10 (sums to 100); Tanks used six dimensions with interaction clarity weighted highest.
- R2 → code-measured gates. The coordinator wrote headless Chrome tooling measuring contrast, horizontal overflow, focus on every Tab stop (392 of 392 focused on the author’s site), and reduced-motion behaviour. The author’s preference: “I’d rather those numbers came from code than from a model squinting at a screenshot.”
- R3 → a stagnation counter. The loop script keeps the counter; ties never reset it, and only strict improvement clears it.
- R4 → roster manifest plus claim-to-evidence checks. The roster is verified at selection time — access checks now run at minute five, moved there after day-one lessons. The author’s proposed next step tightens verification further: before a score counts, compare each claim in a judge report against that judge’s own screenshots and dumps. The practice guide’s finding format pairs naturally with this — severity, evidence (screenshot, file, or visible UI detail), issue, user impact, smallest fix, owner (Agentic Design School, Design Critique Loops With Agents).
- Re-score rule. A code-review fix cannot inherit design approval; the repaired build must be re-scored by every judge before it counts.
What the Four Runs Show
Of the four runs, three ended in merged PRs (#34, #148, #123) and one — Tanks — stopped on ties and remained unaccepted at publication, per the source’s own table and caveats (Sean Carey’s design-loop write-up).
| Product | Judges (primary; panel) | Baseline | Final | Result |
|---|---|---|---|---|
| Tanks (fuel prices) | Gemini 3.8 Flash; retry panel Opus 5.5 + Sol (Gemini dropped by operator) | 6.5 | Gemini 9.0 in rounds 1–3; retry: Opus 7.5 / Sol 8.0 start, both 9.0 at repair 10 but screenshot QA failed; repair 11 in judging at writing time | No PR; not accepted |
| typing.football | Sol; Muse then Gemini | 7.3 | 9.0 from both | PR #34 merged |
| FrugalBar (macOS menu bar) | Sol; Muse then Gemini | 6.3 | Sol 9.0, Gemini 9.1 | PR #148 merged |
| This site (seankoji.com) | Sol; Muse then Gemini | 7.3 | 9.0 from both | PR #123 merged, released 0.9.0 |
Sol = OpenAI gpt-6.1-sol; Opus = Claude Opus 5.5; Gemini = Gemini 3.8 Flash; Muse = Muse Wonder Full 1.0. Scores are on the source’s 6–10 scale. The target was 9.0 in every run; the agent suggested 8.5 in two of them.
- The Tanks stop rule, step by step. Round 1 at 9.0 failed the CSS and JavaScript size limits. Round 2 at 9.0 failed browser tests — a tie. Round 3 at 9.0: the judge’s own action log showed no coverage of station tabs or keyboard chart access — a second tie. “Two ties end the run,” and it did. On the retry, Opus took nine repairs to reach 9.0 while Sol was at 9.0 from the second repair; at repair 10 both sat at 9.0 but screenshot QA failed, and repair 11 was still in judging at writing time. The author admits Sol was judging code its own lineage wrote.
- Judges disagree on identical inputs. typing.football round 3: Sol scored 9 on all seven dimensions; Muse scored 7/7/6/7/6/6/8 (mean 6.7, the source’s table value, design-loop) on the same build under the same rubric. On Tanks, Opus kept consistency below 9 for nine rounds, citing a table collapse on page 1, a changed shelf hint, sheet height, and focus differences; in the panel retry it scored 7 where Sol gave 9.
- The strictest judge kept running out. Muse “was the strictest judge I had” and ran out of provider quota in three of four runs; each swap installed Gemini 3.8 Flash, whose scores sat closer to Sol’s and who “was sometimes wrong about what was on the screen.”
- Gemini’s false claims, caught by the author. “99.99% availability” and a homelab “rack diagram” on this site (neither exists); a hint “far below” a button its own screenshot shows directly underneath; a screen-reader finding marked “Fixed” when nothing changed; zero unnamed controls on FrugalBar while its own accessibility dump listed one; “trapped” Tab focus that was not trapped; a Tanks warning banner absent from the screenshot. “The screenshots were genuine every time. The reports describing them weren’t.” And the operational warning: “The panel-minimum rule only protects you while the strict judge stays on the panel. Replace it mid-run with a more forgiving one and the bar quietly drops.”
- Measurement caught what scores missed. Sol, one judge, measured orange buttons at 3.58:1 on typing.football and a “Configured” label at 1.81:1 on FrugalBar, both below 4.5:1, and found a 9 px badge and a button 16 px below the fold.
- Strictness paid. Muse found match commentary present only in pitch-image alt text (design-loop) — invisible to sighted children — and a “Please select one of these options” tooltip on an already-selected picker; Muse wrote “‘Fail’ is the harshest word in the product, aimed at the slowest typists” and marked efficiency, visual craft, and emotional appeal down to 6 on a build Sol scored 9 across the board. Sol gave FrugalBar visual craft 8 on six builds because warning symbols on long charts “merge into dense bands”, and caught “1 trophies”, fixed to “1 trophy in your cabinet”.
- Code review caught what both judges missed. An already-accepted typing.football design was blocked in independent review: a new single-key shortcut — Gemini’s ask — fired wherever focus was, failing WCAG 2.1.4 (Character Key Shortcuts). Both judges had scored it 9.0. “A screenshot can’t show that bug.”
- The stop rule evolved. On day one the skill demanded a round cap or spend ceiling; when the operator said “unlimited”, the agent tried to set a deadline instead. The current rule relies on the stagnation stop.
Limitations: Read the Confidence Correctly
The evidence base is a single author, four runs, no control arm, model judges rather than users, and one run that never reached acceptance — limits the source states itself (Sean Carey’s design-loop write-up).
- The series is one author’s four runs with no control arm; Tanks never reached acceptance; the judges are model outputs, not users; the scores are judgments.
- The only controlled evidence in scope is a different tool. CritiqueCrew ran two studies totaling N=48 and reported significantly improved design quality and subjective experience versus a static checker in Figma, crediting the structured orchestration of expert roles — not a unified model — as key (CritiqueCrew).
- For ground truth, contrast SimAB: persona-conditioned agents scored against 47 historical A/B tests with known outcomes reached 67% overall accuracy and 83% on high-confidence cases (SimAB). The design-loop runs have no such ground truth. Related work such as PerceptUI, an EMNLP 2026 industry-track paper on persona-conditioned proxy evaluators, is noted by name and venue only (PerceptUI).
- Not established by this evidence: whether the rules generalize beyond one author and one stack; whether model-judge scores in this loop correlate with human user judgments; whether the 9.0 target maps to user value; why Gemini’s reports diverged from its screenshots; and any model-quality ranking beyond the source’s stated observations.
FAQ
These answers draw on the four-run series, a practice guide, and the one controlled study in scope — CritiqueCrew, N=48, which found structured multi-role critique beat a static checker in Figma (CritiqueCrew).
1. Why should a tie count as no progress? In the published series the Tanks run scored 9.0 in three rounds, and each of those rounds failed a gate; the rule that two rounds in a row without strict improvement stop the run ended it. Counting ties as progress would have let a flat score continue past failing gates.
2. What makes a judge “independent”? The primary judge must come from a different model family than the code author. Judges are read-only, must drive the real UI in a browser and bring fresh screenshots, never see each other’s scores, and cannot read code instead of the product. One same-family judge at a higher reasoning level is allowed.
3. Can a 9.0 score override a failed contrast check? No. Measured checks — contrast, keyboard focus, horizontal overflow, reduced motion, repository tests — run beside the rubric as gates, and “a high score can’t waive a failed check.” Sol measured orange buttons at 3.58:1 on typing.football, below the 4.5:1 threshold, and a judge’s overall score does not change that measurement.
4. Is model-judged critique user research? No — the source says “It isn’t user research, and the PRs never pretend it is.” Judges are model outputs writing about screenshots. The only controlled study in scope, CritiqueCrew (N=48, in Figma), tested structured multi-role critique against a static checker, not this loop against real users.
5. What if the strictest judge drops out mid-run? Muse, “the strictest judge I had,” ran out of provider quota in three of four runs and was swapped for Gemini, whose scores sat closer to Sol’s. The author’s warning stands: the panel-minimum rule only protects you while the strict judge stays — swap in a forgiving one and the bar quietly drops.
The Bottom Line
The rules the four runs exercised are an independent judge you cannot talk past, gates a score cannot waive, and a stop rule ties cannot game (Sean Carey’s design-loop write-up).
Three of the four runs shipped as merged PRs. The fourth (Tanks) stopped on two ties, was restarted with a new panel, and was still unaccepted at publication. The strictest judge produced the best feedback and was the one that kept running out of quota, and measured checks caught defects that 9.0 scores had already blessed. On the scale of evidence: one author, four runs, model judges. Adopt the rules, not the confidence.
How This Guide Was Built
This guide is built from the published write-up and the papers cited inline; this blog did not run the tools hands-on (Sean Carey’s design-loop write-up).
It rests on the published write-up (dated 2026-11-04, fetched 2026-11-14, saved in full), the cited practice guide, and the cited papers. This blog did not run the design-loop skill, the judges, or any hands-on test. Every number comes from those sources; anything beyond them is marked not established.
