Getting started

Quickstart

Available

Backtest any public repository in one command. No account, no server — the CLI talks to GitHub, caches locally, and prints the full scoreboard.

Requirements

Python 3.14+ and uv
A GitHub token — picked up from the environment or the gh CLI. Without one you get the unauthenticated 60 req/hr limit, which is only enough for tiny runs.

What a run does

Labels defect-inducing PRs from revert anchors in git history (dense, zero API quota)
Harvests merged PRs into a resumable local cache (.backtest-cache/)
Replays Doug’s scoring over the window and prints capture, cleared-band, per-rule, and holdout tables

Tip — dodge right-censoring:

Young PRs haven’t had time to be reverted yet. Pass --before with a date at least a few weeks back so every PR in the window had a fair chance to fail.

# install (from the repo, for now) $ git clone https://github.com/coldworkshq/doug && cd doug/api $ uv sync # backtest a year of history $ uv run doug-backtest getsentry/sentry \ --limit 5000 --before 2026-06-15 --labels git

5000 scored PRs · 67 in-window labeled defect-inducing flag rate doug size-only random 10% 14% 8% 10% 20% 36% 19% 20% 30% 57% 31% 30% AUC — doug 0.640 · size-only 0.552 · random 0.500