Evaluations
Compare models, harnesses, and skills against recorded tasks and explicit checks.
An evaluation asks whether a specific change improves the work you need done. Keep a baseline, run the candidate on the same cases, and preserve the evidence for each result. Marbles supplies steps, sessions, artifacts, and triggers; you supply the cases and the checks that judge them.
record cases → run baseline and candidate → check results → inspect differences
↑ │
└── propose one change and repeat ──────┘Prerequisites
Complete the quickstart and authenticate the CLI providers you want to compare. Prepare cases with expected outcomes and a fixed baseline. The skill example lists its dependencies and supplies two starting fixtures.
Decide what you are testing
| Subject | What changes | What stays fixed |
|---|---|---|
| Model | The model identifier. | Harness, tasks, instructions, tools, and settings where supported. |
| Harness | Tool access, context handling, or execution environment. | Model and task set where possible. |
| Skill | Instructions, supporting files, or a selected set of skills. | Model, harness, fixtures, and scoring rules. |
A harness is the software that runs the model’s conversation and tools. A trial is one candidate executing one case. A suite is the collection of cases and scoring rules. Use these names in reports so a “pass” has a clear meaning.
Start with cases you can judge
Use real failures alongside normal tasks and cases where no action is needed. Each case needs an input, a starting revision or fixture, expected results, and a time or cost budget. Keep held-out cases out of the examples used to revise prompts and skills.
Prefer executable checks for observable results. Code should compile and pass relevant tests; a review should identify the seeded defect and avoid inventing one in the clean fixture. Use a person or a calibrated model judge for qualities those checks cannot establish. Keep that judgment separate from command exit codes.
Keep enough evidence to compare
Record the case and suite versions, model and provider, harness version, prompt and skill contents, settings, tool permissions, environment, output, test results, elapsed time, and reported usage. Mark unavailable values as unknown. Start each independent trial in a fresh session and reset any files the previous trial could change.
Agents provides sessions; Models resolves their routes. Artifacts keeps the trial records and comparisons.
Continue with model and harness comparisons, then test a skill and run checks when it changes.