Results¶

n = 1000 (300 bad) · labels: UCI Statlog German Credit labels (1994), recoded 2 = bad · scorer: deterministic code (scikit-learn metrics on pooled out-of-fold predictions) · model: champion-lr-sigmoid-v1 · run date: 2026-10-29 · uncertainty: percentile bootstrap, 2000 row resamples, 95% · validation: repeated stratified 5-fold CV x 5, seed 20261029

Model¶

Trivial baselines: approve everyone 1.50, decline everyone 0.70 per applicant.

Challenger minus champion (paired bootstrap): AUROC 0.012 [-0.003, 0.029]; expected cost -0.004 [-0.053, 0.043].

Policy: threshold 0.1667, refer ceiling 0.25; out-of-fold band counts approve 339, refer 161, decline 500.

Serving model: fitted on n = 1000; attribution reference approve_band_training_mean (n = 340); calibration slope -0.806 (negative = monotone increasing).

Fairness (pooled out-of-fold, binary policy)¶

sex¶

Proxy audit (binary): AUROC 0.69 [0.66, 0.73].

foreign worker¶

Proxy audit (binary): AUROC 0.75 [0.66, 0.82].

age band¶

Proxy audit (macro one-vs-rest): AUROC 0.73 [0.70, 0.75].

Verifier¶

n = 1006 genuine notices · labels: designed by construction (mutations of genuine notices) · scorer: deterministic code · model: champion-lr-sigmoid-v1 · run date: 2026-10-30 · uncertainty: a finite test count, not an error-rate estimate

False rejects of genuine notices: 0.

Meta-mutation: 8 of 8 checks, when disabled, made the gate fail.

Analyst guard rails¶

n = 19 cases · labels: designed by construction (single author) · scorer: scripted stand-in (deterministic code around a replayed model) · model: n/a (stand-in) · passed 19 / 19. This measures the code around the model, not a live model's answers.

Analyst live-model run (not gated)¶

n = 15 questions · labels: designed by construction (single author); expectations are tool calls and refusals, not answer quality · scorer: deterministic code checks on tool calls and refusals, plus the analyst's own grounding check; no human or LLM judge of answer quality · model: gpt-4o-mini · run date: 2026-10-30 · uncertainty: n = 15 is too small for a confidence interval; read these as a smoke test of one model, not a rate

Behaved as expected (right tools, refusals, approval held): 15 / 15. Of 12 answers containing numbers: 10 verified first try, 1 after one repair, 1 flagged unverified. Median 4.8 s per question.

For comparison, the first live run before the fixes described in the case study: behaved as expected 8 / 15; of 5 answers with numbers, 1 verified first try and 4 flagged; median 7.4 s.