Build and run evals
An eval case is a question with its approved SQL. Rerun the cases whenever the context changes, so a fix for one answer does not quietly break another.
Save an approved answer
Once a fix is published and checked, save its question as a case, so a later change cannot break it unnoticed. In the running example, that is “How many customers are currently active?”, with SQL that counts customers with a delivered order placed in the 90 days ending at the dataset’s latest order purchase timestamp. Have the owner of the definition validate the SQL first; a validated case is an approved case.
In a git-managed project:
Save the active-customers question and SQL we just approved as a Cassis eval case.SELECT COUNT(DISTINCT c.CUSTOMER_UNIQUE_ID) AS active_customers
FROM STALLORA.CUSTOMERS c
JOIN STALLORA.ORDERS o ON o.CUSTOMER_ID = c.CUSTOMER_ID
WHERE o.ORDER_STATUS = 'delivered'
AND o.ORDER_PURCHASE_TIMESTAMP >= DATEADD(
day, -90,
(SELECT MAX(ORDER_PURCHASE_TIMESTAMP) FROM STALLORA.ORDERS)
)Command: cassis eval add-case -q "<question>" --gold-sql-file gold.sql. The case is stored in Cassis, not in the repository.
In an app-managed project:
On the answer in the chat, choose Add as test case. To write one from scratch, open Evals → Test Cases, choose Create test case, fill Question and Gold SQL, then Create.
The case joins the project’s shared suite right away. Adding it publishes nothing.
Run the cases before publishing
In a git-managed project:
Run the Cassis eval cases against my local changes.12 cases against your local changes: 12 ✓, 0 ✗. Accuracy 100%.
Run 01a05c58.
Agent runs: cassis eval run
Example agent: 12 cases against your local changes: 12 passed, 0 failed. Accuracy 100%. Run 01a05c58.
Command: cassis eval run, which scores the local files without publishing them. Add the same command to CI on pull requests, so a regression blocks the merge (CI setup).
In an app-managed project:
Nothing runs them on their own. In the Publish dialog, choose Run evals to score exactly the changes you are publishing. To run them earlier, choose New run in Evals, on main or on the Cassis branch that holds your edits. Every run lands in Evals → Runs:
| Date | Status | Result | Version |
|---|---|---|---|
| Jun 6, 14:10 | Completed Repeat ×3 | 11/12 agreed | v14 |
| Jun 6, 11:45 | Completed | 92% | v14 |
| Jun 5, 15:20 | Completed | 83% | v13 |
| Jun 2, 10:30 | Completed | 75% | v12 |
With a connected warehouse, Cassis executes the generated and expected SQL, then compares their result rows deterministically, with tolerance for insignificant numeric differences. On a schema-only project, where Cassis has no live warehouse, neither query can run. An LLM equivalence judge compares the SQL logic and explains whether it matches. Passing covers the approved cases, not every possible question.
When a case fails
Say a later change makes the active-customers case fail. A failed case in an ordinary run goes to the issue queue with its evidence. Open the result:
In a git-managed project:
Show me why the active-customers case failed in the latest eval run.“How many customers are currently active?” failed in run 01a06f2e.
- Rows: expected 3,912, got 5,207
- Tables read: CUSTOMERS, ORDERS, the same as the gold SQL
The generated SQL dropped the delivered filter:
- WHERE o.ORDER_STATUS = 'delivered'
- AND o.ORDER_PURCHASE_TIMESTAMP >= DATEADD(
+ WHERE o.ORDER_PURCHASE_TIMESTAMP >= DATEADD(
day, -90,
(SELECT MAX(ORDER_PURCHASE_TIMESTAMP) FROM STALLORA.ORDERS)
)
The gold SQL matches the agreed rule, so the context needs the fix.
Agent runs: get_eval_run (MCP)
Example agent: How many customers are currently active? failed in run 01a06f2e. Rows: expected 3,912, got 5,207. Tables read: CUSTOMERS, ORDERS, the same as the gold SQL. The generated SQL dropped the delivered filter: the gold SQL has WHERE o.ORDER_STATUS = 'delivered' AND o.ORDER_PURCHASE_TIMESTAMP >= DATEADD(day, -90, (SELECT MAX(ORDER_PURCHASE_TIMESTAMP) FROM STALLORA.ORDERS)); the generated SQL has WHERE o.ORDER_PURCHASE_TIMESTAMP >= DATEADD(day, -90, (SELECT MAX(ORDER_PURCHASE_TIMESTAMP) FROM STALLORA.ORDERS)). The gold SQL matches the agreed rule, so the context needs the fix.
Commands: get_eval_run over MCP, or cassis eval history <case-id>.
In an app-managed project:
In Evals → Runs, open the run, then the failed case. Footprint lists the tables each query reads.
The rows differ, and the SQL comparison shows why: the generated query dropped the filter on delivered orders. Both queries read the same tables, so the agent found the right data and applied the wrong rule. On a schema-only project, the judge names where the generated SQL first departs instead: table, join, filter, aggregation, time window, or grain.
The gold SQL still matches the agreed rule, so the context is what needs fixing. Fix it through your workflow, then rerun.
Update or remove a case when the approved business definition changes, never to make a run pass. A stale expected query makes the suite noisy and easier to ignore.
When a case flips
A case that passes on one run and fails on the next, with nothing changed, cannot be trusted to block a merge. Cassis runs each new or edited case three times and marks it Flaky when the attempts disagree, or Failing when none passed, on the Test Cases tab and in cassis eval list-cases. A repeat run files nothing in Review. Different SQL across attempts usually means the context leaves room for two readings, such as a metric description that does not say which orders count: tighten it, then recheck.
In a git-managed project:
Run the active-customers eval case 3 times and show me how the attempts compare.Commands: cassis eval run --case <case-id> --repeat 3, cassis eval compare and cassis eval history. The agent finds the case ID with cassis eval list-cases.
In an app-managed project:
In New run, choose 3 or 5 attempts per case. The run shows whether the agent wrote different SQL or the judge ruled differently. On the case’s page, select two results to compare their SQL.