Prompt Eval Harness

Problem

Prompt changes ship on vibes. A tweak that helps one case silently regresses ten others, and nobody notices until users do.

Approach

Treat prompts like code: every change runs against a golden dataset and reports a score diff before merge.

const result = await runEval(suite, { model: "claude-opus-4-8" });
expect(result.score).toBeGreaterThanOrEqual(baseline.score);

• Deterministic scoring with per-case breakdowns.
• Regression gate wired into CI on every pull request.

Results

Early days, but already caught two regressions that would have shipped. Building out the dashboard and multi-model comparison next.