Evals
Read an eval run's status and scored per-case results. Run ids come from issue occurrences sourced from failing evals, or from runs started in CI or the web app.
Parameters: project_id (optional with an API key scoped to one project, which it defaults to), run_id.
| Field | Notes |
|---|---|
status | "running", "completed", "failed", or "cancelled" |
label | The run’s label on the Evals page. CI runs are labeled with the branch name |
total_cases, completed_cases | Progress counters, in cases. On a repeat run a case counts once, however many attempts it ran |
summary | Passed and failed counts plus accuracy. On a repeat run they count attempts. Null until the run finishes |
repeat_count | Attempts per case on a repeat run (2 to 5), null on an ordinary run |
purpose | "entry_check" on the three-attempt check Cassis runs on a new or edited case, null on a run someone started |
repeat | On a repeat run or an entry check only: {cases, all_passed, all_failed, flipped, causes, flipped_cases} |
results | [{question, status, attempt?, duration_seconds, plan_seconds, execution_seconds, dependency?, error, sql?}] |
error | Set instead of the keys above when the request fails |
Results are written together when the run finishes. Poll status while it is running; do not wait for partial results.
Every case carries how long planning and execution took, and dependency when the run recorded it: state (matched, extra, diverged, or unknown), missing, and extra, comparing the tables the gold SQL reads with the tables the plan used. Each non-passing case includes the generated sql, gold_sql, expected_row_count, actual_row_count, missing_concepts, judge_verdict, judge_divergence (where the generated SQL first departs from the gold SQL: wrong_table, wrong_join, wrong_filter, wrong_aggregation, wrong_time_window, wrong_grain, or other), and judge_reasoning when available. Result row values and the process log are not included. The question and gold SQL come from what the run scored; older results fall back to the current test case.
Repeat runs
A repeat run, started with cassis eval run --repeat N or with 3 or 5 attempts in the app’s New run panel, runs every case several times on the same context and model to measure noise. It opens no issue and is not the run shown before publishing. Each result carries its attempt number. repeat compares the attempts per case: how many cases passed every attempt, failed every attempt, or flipped, and for each flipped case its question, passed out of total, distinct_sql (how many different SQLs the agent wrote), judge_unstable (the judge ruled differently on one SQL), and cause:
cause | Meaning |
|---|---|
generation | The agent wrote different SQL from the same context: the context leaves room for both |
judge | The judge ruled differently on the same SQL: a gold SQL or question-wording problem, not a context gap |
both | Both moved |
unattributed | The evidence decides neither side |
What it cannot do
There is no tool to start a run, add a case, or delete one. Runs start from CI or a checkout with cassis eval run, or from the Evals page in the app. An agent that has just fixed an issue therefore runs the CLI in its checkout, and uses this tool to read a run someone else started.
An eval run scores a version of the context. The context an agent is changing lives in its checkout, so the CLI starts that run.