MCP servers · sadri-dridi

Agent Crawl Readiness (MCP server)

An MCP server that runs a bounded crawl-readiness check of a website for agent access, covering robots.txt, sitemap.xml, llms.txt and the homepage.

This text was machine-translated from the publisher's own information and has not been checked by a human editor. The publisher's original text is the reference.

This resource was imported automatically from the sources listed below and has not been reviewed by the AI Gate team yet. The description is the publisher's own, as published. Check the repository before installing.

What it does

The server checks four parts of a website that affect how agents can access it: robots.txt, sitemap.xml, llms.txt and the homepage. The check is bounded, so it covers only these items rather than a full crawl.

It suits anyone who needs to see how a site presents these files to agents before agents access it. The publisher's documentation provides no further details on setup or running requirements.

Safety

Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.

  • Runs code or changes an assistant's behaviour

Sources

  • Official MCP Registry first seen 13 Nov 2026 · last seen 13 Nov 2026 · its page on the source
    Agent Crawl Readiness

    Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

    v1.0.0
    As shown on registry.modelcontextprotocol.io when we last read it
MCP servers
hashicorp

Terraform MCP Server (HashiCorp)

HashiCorp's MCP server that gives an assistant current Terraform Registry information on providers, modules and policies, and can manage HCP Terraform workspaces.

1.6K GitHub starsCloud & DevOpsSoftware development
Updated 17 Nov 2026
MCP servers
modelcontextprotocol

MCP reference servers

Seven small example servers maintained by the Model Context Protocol project: files, Git, web fetching, memory, time, step-by-step thinking and a test server.

92K GitHub starsSoftware development
Updated 18 Nov 2026
MCP servers
redis

Redis MCP Server

Redis's official MCP server that lets an AI assistant store, read and search data in a Redis database, including streams, JSON documents and vector indexes.

642 GitHub starsData & databasesSoftware development
Updated 17 Nov 2026
MCP servers
firecrawl

Firecrawl MCP Server (web search and scraping)

Firecrawl's MCP server that lets an AI assistant search the web, turn pages and PDFs into clean Markdown, crawl sites and interact with dynamic pages.

7.8K GitHub starsWeb, search & browserProductivity & automation
Updated 17 Nov 2026
Agent Crawl Readiness (MCP server) | AI Gate Discover