Compare models and harnesses
Make controlled comparisons and keep failures, uncertainty, and operating cost visible.
Start with a question you can answer, such as whether a candidate finds more real review defects without increasing false alarms. Changing the model, prompt, skill set, and tool permissions together tests the whole setup. It does not tell you which change helped.
Freeze the comparison
Select a baseline and candidate. Pin the fixture revision, suite, and scoring rules before running either. Record provider and model settings; some providers may not support equivalent settings. A CLI provider also brings its own instructions and tools, so describe comparisons across providers as comparisons of complete setups.
Use a fresh session per trial. For code-writing tasks, create a fresh checkout or sandbox and run the same verification commands afterward. Keep expected answers and the grader outside the candidate’s accessible files when testing whether it can solve a task.
Score the work
| Evidence | What it answers |
|---|---|
| Tests and required commands | Did the candidate meet the checked requirements? |
| Defects found and false alarms | Did the reviewer identify real problems without inventing new ones? |
| Human assessment | Is the code maintainable and appropriate for the original intent? |
| Duration and reported usage | What did the result cost to obtain? |
| Errors and timeouts | Could the setup complete the task within its budget? |
Keep errors in the denominator. Distinguish a provider outage from a wrong answer, but do not silently discard failed trials. A repair attempt is another trial with its own cost; an automatic workflow retry can hide that distinction. The durability guide explains retry and replay.
Repeat cases when responses vary. Compare per-case regressions as well as the overall score. Save transcripts and candidate revisions so a person can inspect why a score changed. If a model grades the output, record its version and rubric and check a sample of its judgments against human decisions.
Decide whether to adopt
Set the acceptance rule before reading candidate results. For example, require no regressions on critical cases and fewer unsupported review findings across the suite, within an agreed runtime budget. Record the sample size and the remaining gaps alongside the decision.
Keep the previous setup available. Changing a provider’s default model can change results even when your code is unchanged; record the resolved model when available and schedule a periodic comparison. An alias alone is not an immutable version.
The skill example runs a small local comparison through the current CLI providers. Larger suites need their own fixtures, isolated execution, and grading. Marbles does not include a model leaderboard or automatic promotion policy.