doug
DashboardScoreboardQueueDocsGitHubAbout
Menu
DashboardScoreboardQueueDocsGitHubAbout
Sign in

Getting started

  • Introduction
  • Quickstart

Concepts

  • Risk routing
  • Defect labels
  • The cleared band
  • What Doug gets wrong

Reference

  • CLI · doug-backtest
  • The report

Coming up

  • MCP · Pattern Garden
  • REST API

Meta

  • Changelog
View as llms.txt

Concepts

Defect labels

Available
A backtest is only as honest as its labels. Doug labels a PR defect-inducing when later history reverts it — the revert is an engineer’s own on-the-record verdict that the change was bad.

Label sources (--labels)

git default
Parse revert anchors out of git history directly. Dense, zero API quota, and reproducible from a clone alone.
api
Search GitHub for revert PRs via the API. Sparser but catches reverts that never say "revert" in the commit subject.
both
Union of the two. Labels are intersected with the harvested window — no outcome-dependent injection, which would bias the capture numbers.

Known limits, stated plainly:

Reverts under-count defects (hotfixes without reverts are missed), and ~6% of git labels on sentry show timestamp anomalies we’re still triaging. The labels are good enough to rank on; we publish their weaknesses along with the wins.
How a label is born
# merged PR #4821 — "migrate retry-queue table"
merge  2026-03-02  a41f9c2

# nine days later:
commit 2026-03-11  e77d201
"Revert 'migrate retry-queue table' (#4821)"     

→ #4821 labeled defect-inducing
# an engineer said so, in the record, at the time
PreviousRisk routingNext The cleared band