doug
DashboardScoreboardQueueDocsGitHubAbout
Menu
DashboardScoreboardQueueDocsGitHubAbout
Sign in

Getting started

  • Introduction
  • Quickstart

Concepts

  • Risk routing
  • Defect labels
  • The cleared band
  • What Doug gets wrong

Reference

  • CLI · doug-backtest
  • The report

Coming up

  • MCP · Pattern Garden
  • REST API

Meta

  • Changelog
View as llms.txt

Honesty

What Doug gets wrong

Available
Doug reviews every pull request in his own repository, and we write down what he got wrong. Just under a third of prospective findings are disproved by code he wasn’t shown — he reads a diff, not a repository, and he does not reliably distinguish what the diff proves from what it merely permits.

Two failure classes recur often enough to have names:

  • A conclusion drawn from an absence in the diff. “This name is used and no import was added” — when the import was already there, three files away.
  • Re-reporting a tradeoff the code already documents, as though it were news.

The first one produced a rule worth more than the bug it came from: a claim about an absence cannot be settled by looking at the same place the claim came from. “No import was added” is a fact about the diff; whether the import exists is a fact about the repo. Re-reading the diff confirms the finding every time and proves nothing — the check and the error are the same observation. In that particular case the linter had already answered it, green, before Doug ever spoke.

So every finding now gets a line at disposition time: was it real, disproved, or adjacent — wrong as stated, right about something nearby — and separately, did anything in the codebase actually change because of it.

Two axes, deliberately, because one column loses the cases that matter. A true finding that changed nothing is a re-report of something the code already says. A false finding that changed something found a real gap by the wrong route. Doug’s strongest mode is “this code does not justify itself” — and a single score would grade that as failure.

There is no rate here yet, and that is the point.

Backfill was reconstructed from write-ups after the fact and is excluded from every rate by construction; the denominator is the prospective rows, scoped to one repository and one instrument, because a rate computed across two of either describes neither. The counts beside this paragraph are a dated snapshot, not a live counter — the log grows every time a finding is settled, and a number printed here cannot follow it. Read the current one off the log yourself; it ships in the repo and the command is in the panel. When there is a rate, it will not be called precision: whether a finding is true is a different quantity from whether it predicted a defect, and a finding can be true and worthless or false and load-bearing.
FINDINGS LOG · ONE LINE PER FINDING
{"pr": 28,
 "rule": "reader:missing-import",
 "verdict": "disproved",
 "changed": false,
 "settled_by": "api/doug/api.py:7 —
   already imported; ruff F821 was
   green before the finding"}

# as of 2026-08-27 — a snapshot, not a
#   counter. 205 rows; 193 prospective,
#   12 backfill excluded from every rate.
#   The reader on doug is 176 of those:
#   54 disproved, 85 real, 37 adjacent.
#   The plan lane writes here too, in its
#   own vocabulary, and is not in that
#   figure.

# today's number, from the log itself:
# python -m doug.findings_log rate \
#     --repo doug --rule-prefix reader:
PreviousThe cleared bandNext CLI · doug-backtest