What it does
ConKurrence runs an evaluation schema across several AI raters and measures how far they agree, using Fleiss' kappa with bootstrap confidence intervals and Kendall's W against expert-labelled anchor items. Multi-model runs cover Bedrock, OpenAI and Gemini models, and a self-consistency mode uses the host model through MCP Sampling. Tools cover running evaluations, generating markdown reports, comparing two runs, tracking agreement across runs, suggesting a schema from data, validating a schema and estimating cost and token usage.
It suits cases where you need to know which items are contested and so need an expert's judgment, or whether disagreement comes from the criteria or from the raters. Agreement shows consistency, not correctness: models can agree confidently on a wrong answer, so the tool does not replace expert-labelled ground truth.
It runs as an MCP server in Claude Desktop or any MCP-compatible client, and is also offered as a Claude Code plugin. The self-consistency mode needs no API keys, while multi-model runs depend on access to the Bedrock, OpenAI and Gemini models. Cost can be estimated before a run.
Limitations and requirements
The project is in maintenance mode: it is not accepting purchases and carries no support commitment. Agreement measures reliability, not validity. Multi-model runs depend on access to Bedrock, OpenAI and Gemini models. Licensed under BUSL-1.1 (Business Source License 1.1).
Safety
Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.
- Runs code or changes an assistant's behaviour
Sources
- Official MCP Registry first seen 13 Nov 2026 · last seen 13 Nov 2026 · its page on the source
io.github.AlligatorC0der/conkurrence
Multi-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.
v1.0.5As shown on registry.modelcontextprotocol.io when we last read it