What it does
evalsnap compares two snapshots of case outputs, a baseline run and a current run, matched by case_id. For each matched case it reports whether the output is unchanged or changed under exact string equality, along with the score delta and duration delta. Either delta is null when one side did not record that measurement. A case is flagged as a regression only when its score went down. The result also lists added and removed cases, the sorted regressed case ids, the count of improved cases, and averages for each side over the matched cases that have a score on both sides. The verdict is fail when at least one regression exists and pass otherwise. Added and removed cases are reported but do not fail the verdict on their own. Deltas and averages are rounded to four decimal places.
It is useful after re-running an agent or tool when the answers look slightly different and you need to know whether any scores got worse. It is available as a command-line tool that prints JSON, and as an MCP server exposing a diff_snapshots tool.
The computation is local and pure: it makes no network calls, needs no configuration, and reads only the snapshots you pass in. The CLI takes an args.json file holding the full arguments, and exits with 0 on success, 1 on a runtime error, and 2 on a usage error. The MCP tool takes baseline and current arrays of entries shaped as case_id, output, and optional score and duration_ms.
Limitations and requirements
Invalid input is rejected with an error naming the offending path and side, and no partial diff is produced. Rejected input includes an empty snapshot, a case without an id or output, a duplicate id within one side, a non-numeric score, and a negative duration. Running it through an MCP client requires adding a client configuration entry.
Safety
Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.
- Runs code or changes an assistant's behaviour
Sources
- Official MCP Registry first seen 13 Nov 2026 · last seen 13 Nov 2026 · its page on the source
io.github.CloudCorpRecords/evalsnap
Snapshot diff for agent and tool outputs: changed cases, score regressions, pass/fail verdict.
v0.1.0As shown on registry.modelcontextprotocol.io when we last read it