Eval commands

eval run

Runs the project’s eval suite and prints per-case results. Failed cases include generated and gold SQL, row counts, missing concepts, and the judge’s verdict and reasoning when available. With --case, passed cases print the same evidence, so you can check that a context change makes a case pass for the right reason; a run of the whole suite lists passed cases on one line. Row values are available in --json and the app, not in terminal output. Generated SQL can contain literals read from your data, so handle CI logs with the same care as queries. Long evidence is explicitly marked as truncated. By default, the CLI uploads the local context as ephemeral run input; it does not publish or replace the project’s context. Pass --branch to score an existing Cassis branch instead, in which case local files are not uploaded.

# Score the local files and wait for results
cassis eval run --project ...

# Run only specific cases (repeatable)
cassis eval run --project ... --case 019f0000-... --case 019f0001-...

# Run against an existing Cassis branch instead of the local context
cassis eval run --project ... --branch feature-x

# Start the run and return immediately, poll in the web app
cassis eval run --project ... --no-wait

# Run every case 3 times and compare the attempts
cassis eval run --project ... --repeat 3

Subset: --case <id> (repeatable) runs only the named cases. Use it to prove one freshly added case in seconds instead of rerunning everything. Ids come from list-cases.
Run input: Omit --branch to upload and score the local context without publishing it. --branch <name> selects an existing Cassis branch and ignores the local tree. An unknown name exits 2 (usage error); use --label to name a local run after your git branch.
Repeat: --repeat N (2 to 5) runs every case N times on the same context and model, then prints how the attempts compare per case instead of the per-case list: see eval compare. A repeat run measures noise: it opens no issue in Review and is not the run the app shows before publishing. It exits 1 only when a case failed every attempt; a case that flipped is the noise being measured. --json adds the comparison under comparison. Combine it with --case to measure one case. Needs cassis-cli 3.4.
Run label: Local runs are labeled with your git branch name on the Evals page, taken from the CI environment or checkout. --label overrides it, and is rejected with --branch, whose runs use the Cassis branch name.
Waiting: --wait or --no-wait, --poll-interval (5 seconds), --timeout (30 minutes). The run keeps going server-side if the CLI stops waiting. Ctrl-C cancels the run and exits 130.
JSON: --json prints the final run and its results as one JSON document on stdout; the started/progress lines go to stderr, so cassis eval run --json | jq works (cassis-cli 1.7).
Deep link: It prints a link to the run’s page in the Evals UI. --app-url or CASSIS_APP_URL overrides the base URL when the web app is not served from the API host.

eval compare

Prints how the attempts of a repeat run compare, case by case: the same comparison eval run --repeat prints when the run ends, and the run’s page in the app shows.

cassis eval compare 019f0000-0000-7000-8000-00000000aaaa --project ...

Summary: How many cases passed every attempt, failed every attempt, or flipped, with the flips counted by cause.
Each case that did not pass every time: Its attempts that passed, how many distinct SQLs the agent wrote, the cause, and the case id.
Cause: generation: the agent wrote different SQL from the same context, so the context leaves room for both. judge: the judge ruled differently on the same SQL; check the gold SQL and the question’s wording. both: the two at once. unattributed: the evidence decides neither side.
JSON: --json prints the comparison record as the server returns it.
Exit: 0 once the comparison is printed, including for a run still in progress, which it says. 3 on a transport or API error, including a server too old to compare runs.

eval add-case

Adds an approved question and SQL case to the project’s suite. Before running it, have a domain owner or reviewer validate the SQL and the business assumptions it encodes. Executable SQL alone does not establish correct business meaning.

cassis eval add-case --project ... \
  -q "What is the average order value?" \
  --gold-sql "SELECT AVG(ORDER_VALUE) FROM STALLORA.FCT_ORDERS WHERE IS_DELIVERED = TRUE"

# Multi-line SQL from a file, which avoids shell quoting issues
cassis eval add-case --project ... \
  -q "What was revenue in 2025?" \
  --gold-sql-file gold.sql

The gold SQL is run first: On an executable data source, Cassis runs your SQL before storing the case, so a case that cannot execute never enters the suite. This catches execution errors, not incorrect business interpretation.
What it stores: The question plus the gold SQL. On a warehouse-connected project, both generated and gold queries run and their rows are compared deterministically with fuzzy numeric matching. On a schema-only project, an LLM judge compares their SQL logic.
Rejections: Exit 1 on a duplicate question or on gold SQL that does not run. Both print the reason; nothing is added.
Footprint warning: Right after adding the case, it warns when the gold SQL reads tables the context lacks, as list-cases does.
Entry check: Cassis then runs the new case three times on the current context, in the background, and the command says so. A minute or two later, list-cases flags it if the attempts disagree or none passed. Adding a case is never refused because of the check, and the check never blocks your own runs.
No checkout needed: It talks to the project, not to your files, so it takes no path and no --base-path.

eval list-cases

Lists the project’s cases, one id and question per line. The ids are what history and delete-case take.

cassis eval list-cases --project ...

# Resolve the gold SQL against a context branch
cassis eval list-cases --project ... --branch feature-x

# Include the gold SQL and the footprint
cassis eval list-cases --project ... --json

Footprint: Each case’s gold SQL is resolved against the unpublished context. A case whose SQL does not fully resolve is flagged, with the reason under its line: a table the context lacks, a bare name matching several context tables, or SQL the dialect cannot parse. Qualify the name or fix the gold SQL to make it resolve cleanly. Projects without a data source show no footprint.
Entry check: A case whose entry check found it flaky (the attempts disagree) or failing (none passed) is flagged with how many attempts passed. A flaky case’s verdict is noise until its question or gold SQL is tightened; a failing one points at its gold SQL or at a context gap. A case still being checked says so. Editing a case’s question or gold SQL in the app checks it again.
Branch: --branch <name> resolves against a context branch instead. An unknown name exits 2.
JSON: --json adds each case’s gold SQL, its footprint (state, tables, unresolved, ambiguous, shadowed, and error), and its entry_check (state: running, stable, flaky, failing, or not_checked; passed, total, run_id, checked_at).

eval history

Shows how one case has behaved across runs: the same account as the case’s page in the web app.

cassis eval history 019f0000-0000-7000-8000-000000000000 --project ...

# The same record as JSON
cassis eval history 019f0000-0000-7000-8000-000000000000 --project ... --json

Health: A verdict (passing, failing, noisy, or changed across configurations), the pass rate, how many configurations disagree with themselves, average plan and execution time, and the judge’s divergence categories and footprint states seen across results.
Results: Every result grouped by the configuration it ran under (context version or branch, model, Cassis version), newest first, each with its verdict, divergence, footprint state, and timings.
Reading it: Verdicts that differ inside one configuration are noise to act on: fix the case or the context until it is stable. A verdict that changes across configurations is the context or the model moving.
Exit: 1 when the project has no case with that id.

eval delete-case

Deletes one case by the id list-cases prints. Use it to prune a case that is stale or wrong, for instance gold SQL encoding a definition the context has since changed.

cassis eval delete-case 019f0000-0000-7000-8000-000000000000 --project ...

Exit 1 when the project has no case with that id; nothing is deleted.