MCP servers · alligatorc0der

ConKurrence: multi-model rater agreement for AI evaluation

An MCP server that measures agreement among several AI raters to flag contested evaluation items for expert review.

This text was machine-translated from the publisher's own information and has not been checked by a human editor. The publisher's original text is the reference.

This resource was imported automatically from the sources listed below and has not been reviewed by the AI Gate team yet. The description is the publisher's own, as published. Check the repository before installing.

What it does

ConKurrence runs an evaluation schema across several AI raters and measures how far they agree, using Fleiss' kappa with bootstrap confidence intervals and Kendall's W against expert-labelled anchor items. Multi-model runs cover Bedrock, OpenAI and Gemini models, and a self-consistency mode uses the host model through MCP Sampling. Tools cover running evaluations, generating markdown reports, comparing two runs, tracking agreement across runs, suggesting a schema from data, validating a schema and estimating cost and token usage.

It suits cases where you need to know which items are contested and so need an expert's judgment, or whether disagreement comes from the criteria or from the raters. Agreement shows consistency, not correctness: models can agree confidently on a wrong answer, so the tool does not replace expert-labelled ground truth.

It runs as an MCP server in Claude Desktop or any MCP-compatible client, and is also offered as a Claude Code plugin. The self-consistency mode needs no API keys, while multi-model runs depend on access to the Bedrock, OpenAI and Gemini models. Cost can be estimated before a run.

Limitations and requirements

The project is in maintenance mode: it is not accepting purchases and carries no support commitment. Agreement measures reliability, not validity. Multi-model runs depend on access to Bedrock, OpenAI and Gemini models. Licensed under BUSL-1.1 (Business Source License 1.1).

Safety

Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.

  • Runs code or changes an assistant's behaviour

Sources

  • Official MCP Registry first seen 13 Nov 2026 · last seen 13 Nov 2026 · its page on the source
    io.github.AlligatorC0der/conkurrence

    Multi-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.

    v1.0.5
    As shown on registry.modelcontextprotocol.io when we last read it
MCP servers
hashicorp

Terraform MCP Server (HashiCorp)

HashiCorp's MCP server that gives an assistant current Terraform Registry information on providers, modules and policies, and can manage HCP Terraform workspaces.

1.6K GitHub starsCloud & DevOpsSoftware development
Updated 17 Nov 2026
MCP servers
neondatabase

Neon MCP Server (Postgres)

Neon's MCP server for managing Neon Postgres in plain language: create projects and branches, run queries and prepare schema migrations from your editor.

663 GitHub starsData & databasesSoftware development
Updated 17 Nov 2026
MCP servers
redis

Redis MCP Server

Redis's official MCP server that lets an AI assistant store, read and search data in a Redis database, including streams, JSON documents and vector indexes.

642 GitHub starsData & databasesSoftware development
Updated 17 Nov 2026
MCP servers
supabase-community

Supabase MCP Server

Supabase's MCP server that connects an AI assistant to your Supabase project to design tables, run queries, read configuration and look up documentation.

Data & databasesSoftware development
Updated 17 Nov 2026