# Doug — risk-routed code review > Doug scores every pull request, clears the majority, and routes the risky > few to a human with evidence attached. Never blocks a merge. Status: EARLY > PREVIEW — public surfaces: /queue, /scoreboard, hosted /docs, GitHub App > check runs. The self-serve measurement tool is the backtest CLI. Docs: > /docs · Repo: https://github.com/coldworkshq/doug ## Status legend - AVAILABLE: works today - PREVIEW: exists, gated, in training - PLANNED: described so you can see the direction; does not exist yet ## Quickstart (AVAILABLE) Requirements: Python 3.14+, uv, a GitHub token (env or `gh` CLI; unauthenticated = 60 req/hr). git clone https://github.com/coldworkshq/doug && cd doug/api uv sync uv run doug-backtest getsentry/sentry --limit 5000 --before 2026-06-15 --labels git A run: labels defect-inducing PRs from revert anchors in git history (dense, zero API quota), harvests merged PRs into a resumable local cache (.backtest-cache/), replays Doug's scoring, prints capture / cleared-band / per-rule / holdout tables. Tip: pass --before a few weeks back to dodge right-censoring (young PRs haven't had time to be reverted). ## CLI reference — doug-backtest (AVAILABLE) doug-backtest [flags] - repo (positional, required): owner/repo, e.g. astral-sh/ruff - --limit INT (default 300): merged PRs to harvest - --before YYYY-MM-DD: only PRs created before this date (right-censoring guard) - --labels git|api|both (default git): defect label source - --token STR: GitHub token (default: env, then gh CLI) - --cache-dir PATH (default .backtest-cache): harvest cache; reusable, seeds bigger runs - --output PATH: JSON report path (default: backtest--.json). Always written. - --backfill-details: fetch per-file stats + patch text for cached PRs (~1 req/PR, resumable) Harvests are checkpointed — kill and relaunch with the same command to resume. ## Concepts - Capture curve (AVAILABLE): reading only the top-scored N% of PRs (the budget), what share of defect-inducing PRs lands in that band? capture@budget is the headline routing metric; AUC summarizes across budgets. Every report includes a size-only baseline — if Doug can't beat "biggest diff first," don't pay for Doug. - Hotspots (AVAILABLE): path segments where the repo's own past defects cluster. Live App scoring uses static hotspot segments. Rolling-window learning is backtest holdout only (older half of the harvest). Measured on sentry: 2.34x revert lift. Hotspot vocabulary does NOT transfer between repos. - Defect labels (AVAILABLE): a PR is defect-inducing when later history reverts it. Sources: git (revert anchors in history; default), api (GitHub revert-PR search), both (union, intersected with the harvested window — no outcome-dependent injection). Known limits: reverts under-count defects; a small % of labels show timestamp anomalies under triage. - Per-repository settings (AVAILABLE): each connected repository carries its own flag line (the score at which Doug asks for a human), a PR-comment toggle, and a deep-read toggle. They are on /dashboard/settings, and the flag line is ALSO on the repositories table, one column from the "needs you" count it decides — both surfaces are sanctioned and render one control against one API (ADR-0013, ADR-0019). Deep read off means that repository is scored on structural signals alone and no diff leaves it; on a repository with no flag line of its own it also moves the line Doug bands against, so he asks for a human less often rather than merely differently. - Carrying findings forward (AVAILABLE): when the code a finding cited is byte-unchanged in the next push, Doug carries that finding forward by construction and reports it under "Since ", with a count of his own earlier findings on untouched code that the reader did not mention again. Nothing is marked resolved: evidence that code was edited is not evidence it was fixed. - Cleared band (AVAILABLE): the product claim is about CLEARED PRs. density_lift = cleared-band defect density vs merging blind; <1 means clearing carries information, =1 means merging blind. This is the trust metric; capture@budget is the routing metric. ## The report (AVAILABLE) Printed tables: capture curve (doug/size-only/random + AUC), cleared band (cleared, missed, miss rate, density_lift per budget), per-rule precision (fired/hit/precision/lift — dead rules stay in the table), time-split holdout (hotspots learned on the older half only; needs >=200 PRs, >=8 defects, >=4 per half, else honestly omitted). The JSON does not list missed PRs by number. JSON schema is early-preview; printed tables are the stable contract. ## Doug's own findings (AVAILABLE — the log; there is no rate yet) Doug reviews every PR in his own repository and the dispositions are written down. Just under a third of prospective findings are disproved by code he wasn't shown: he reads a diff, not a repository, and does not reliably separate what a diff PROVES from what it merely PERMITS. Two recurring failure classes: (1) a conclusion drawn from an absence in the diff ("this name is used and no import was added" — it was already imported elsewhere); (2) re-reporting a tradeoff the code already documents. The rule that came out of (1), which generalizes: a claim about an absence cannot be settled by looking at the same place the claim came from. "No import was added" is a fact about the diff; whether the import exists is a fact about the repo. Re-reading the diff confirms the finding every time and proves nothing — the check and the error are the same observation. In that case the linter had already answered it, green, before the finding was emitted. Each finding gets one line at disposition (docs/findings-log.jsonl): verdict real|disproved|adjacent ("adjacent" = wrong as stated, right about something nearby), whether anything in the codebase actually changed, and the file that settled it. Two axes because one loses the interesting cases — a true finding that changed nothing is a re-report; a false finding that changed something found a real gap by the wrong route. Doug's strongest mode is "this code does not justify itself," and a single score would grade that as failure. NO RATE IS PUBLISHED. Backfill is excluded from every rate by construction; the denominator is the prospective rows, scoped to one repository AND one instrument, because a rate computed across two of either describes neither — the diff reader writes `reader:` rows, the plan lane writes `deviation:` / `beyond-ticket:` / `missing-from-pr:` rows, and they are different vocabularies. SNAPSHOT, NOT A COUNTER — as of 2026-08-27 the log held 205 rows: 193 prospective, 12 backfill, and 176 of the prospective rows were the reader on Doug's own repository (54 disproved, 85 real, 37 adjacent). The log grows whenever a finding is settled, so do not quote those integers as current. The current figure comes from the log itself, which ships in the repo: `uv run python -m doug.findings_log rate --repo doug --rule-prefix reader:`. When a number exists it will NOT be called precision: whether a finding is TRUE is a different quantity from whether it PREDICTED A DEFECT, and a finding can be true and worthless or false and load-bearing. ## MCP — Pattern Garden (PLANNED — no server exists yet) An MCP server a coding agent queries BEFORE writing. Serves outcome-anchored patterns distilled from public-repo history: per problem shape, the solution variants people actually write, each carrying its revert record, survival record, and citations to real PRs. Planned tools: doug.ask (describe the change, get variants ranked by outcome evidence) and doug.check (submit a draft diff, get losing-variant matches). Gated on evidence: patterns ship only after outcome deltas replicate across repos. First domain: schema migrations. No dates promised. ## REST API (PREVIEW — public showcase live; tenant queue and receipt live, gated; tenant scoreboard planned) Live, unauthenticated, pinned to the showcase repo: - GET /v1/showcase/queue — open PRs with scores and reasons (what /queue renders) - GET /v1/showcase/scoreboard — prospective counters; miss_rate is null until the pre-registered interval fires. Per-author-type rates are not published. Live, session or token: - GET /v1/queue — tenant-scoped routed queue - GET /v1/prs/:number/receipt — evidence trail, session or receipt:read token Planned: tenant-scoped GET /v1/scoreboard. ## Research results — LLM diff-read probe (2026-07, PREVIEW — probe AUC, not a measurement of the shipped reader) Pre-registered before any run (bars, samples, kill conditions frozen first). Setup: structured LLM reads of PR diffs (title + files + patch only — no author, dates, or outcome), findings stored per PR, evaluated against revert/hotfix labels. - Ranking: the LLM risk score out-ranked every deterministic baseline we have (size sort, shipped rules, RandomForest on Kamei's 14 metadata features) on BOTH repos tested — AUC 0.69 on sentry, 0.67 on grafana. On grafana every deterministic method sits at or below random; the LLM read is the first method in this project that held up on repo #2. - Robustness: re-reading diffs with polarity inverted (added<->deleted) kept only 15-17% of findings — the reader tracks what the change does, not surface diff patterns (the ReDef failure mode did not reproduce). - Distillation of findings into deterministic checks: recurring-pattern coverage passed marginally on sentry, failed on grafana. The pattern vocabulary is shared across repos; the density is not, on one thin-labelled repo. - Caveats, always attached: evaluation used defect-enriched samples (AUC is unbiased there; capture percentages are not population numbers and are not quoted); grafana had 22 evaluation defects (directional, not definitive); one model, one prompt. Scripts: api/scripts/llm_probe.py, api/scripts/rf_kamei.py. ## Honesty notes All published numbers are scoped to the repo they were measured on (sentry unless stated). The deterministic scorer is a single-repo result — its cross-repo replication FAILED (grafana at random), which we publish rather than bury. The LLM diff-read result above is the piece that replicated across both repos. Those AUC numbers are the 2026-07 probe (ADR-0012), not a measurement of the shipped 100k-char reader that is the live scoring path when DOUG_READER=1 and the reviewed repository has not turned its deep read off. Misses are published, not buried. Doug never blocks a merge. What Doug himself gets wrong is written down too (see "Doug's own findings") — including the fact that there is not yet enough of it to quote a rate. A design for an agent review crew with a falsifier-adjudicated finding ledger was reviewed by six independent reviewers in 2026-08 and cut: four of six proposed "agent lenses" were checks that already ran or free platform features, and the ledger's motivating case was one its own mechanism could not fix, because the obvious falsifier restates the error. We publish the designs we kill, too.