Reference

The report

Available

Every run prints four tables; the same run writes JSON to backtest-{owner}-{repo}.json (or --output). The printed tables are the contract to check. The JSON does not list missed PRs by number.

Sections

Capture curve — doug vs size-only vs random, at each flag rate, plus AUC
Cleared band — cleared count, missed defects, miss rate, density_lift per budget
Per-rule precision — every scoring rule: fired / hit / precision / lift. Dead rules stay in the table
Time-split holdout — hotspots learned on the older half only, scored on the newer half; requires ≥200 PRs and ≥8 defects with ≥4 in each half, otherwise it’s honestly omitted

Schema stability:

early preview — field names may still move. The printed tables are the stable contract for now; treat the JSON as versioned-by-commit.

{ "repo": "getsentry/sentry", "prs": 5000, "auc": { "doug": 0.640, "size_only": 0.51 }, "capture": { "0.20": 0.36 }, "cleared_band": [ { "flag_rate": 0.3, "density_lift": 0.61, … } ], "rules": [ { "rule": "hotspot-path", "lift": 2.34, … } ], "holdout": { … } }