dev.workers.mike-tusa.robotsgate/robotsgate

Est-ce que dev.workers.mike-tusa.robotsgate/robotsgate est sûr ?

Au dernier audit MCP Hub, le serveur MCP « dev.workers.mike-tusa.robotsgate/robotsgate » est classé « Non encore testé ». Référencé dans l'annuaire. Seul le pentest comportemental fait foi, et il n'a pas encore été exécuté sur ce serveur. Le verdict est réévalué en continu ; consultez la fiche pour l'état à jour.

Méthode : MCP Hub analyse la description de chaque outil (couche 1) et, pour le verdict fort, exécute le serveur dans une micro-VM isolée avec leurres et réseau bloqué (couche 2). On publie des noms et des compteurs, jamais le contenu brut renvoyé par un outil.

Les 4 outils

• check_site≈ 447 tokens

  Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.

  Paramètres, retour & exemple (2)

  Paramètres

  • url · string · requis
    Site URL or bare hostname, e.g. example.com. Only the scheme and host are used.
  • path · string · optionnel
    URL path to test, starting with "/" (default "/").

  Exemple d'appel

  {"name": "check_site", "arguments": {"url": "texte"}}

  Exemple de retour

  {"content": [{"type": "text", "text": "…"}], "structuredContent": {"url": "texte", "path": "texte", "cache": {}, "final_url": "texte", "redirects": 1, "fetched_at": "texte", "ai_crawlers": "texte", "http_status": 1, "robots_found": true, "interpretation": "parsed", "personal_agents": "texte"}}

• generate_robots≈ 763 tokens

  Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.

  Paramètres, retour & exemple (9)

  Paramètres

  • date · string · optionnel
    Date for the header comment (YYYY-MM-DD). Default: today (UTC).
  • agents · object · optionnel
    Per-token overrides, e.g. {"GPTBot": "block"}. Tokens: letters, digits, "_", "." and "-", up to 64 characters (not "*"). At most 100.
  • default · string · optionnel · allow | block
    Policy for "User-agent: *" (default allow).
  • sitemap · string · optionnel
    A single sitemap URL (alternative to sitemaps).
  • sitemaps · array · optionnel
    Absolute http(s) sitemap URLs. At most 20.
  • categories · object · optionnel
    Action per crawler category. Omitted categories get no group of their own.
  • allow_paths · array · optionnel
    Paths to allow explicitly. Each path starts with "/" or "*" and has no whitespace or "#". At most 100.
  • custom_rules · string · optionnel
    Extra robots.txt lines appended after validation (max 20000 bytes).
  • disallow_paths · array · optionnel
    Paths to disallow for "*" and for allowed AI groups. Each path starts with "/" or "*" and has no whitespace or "#". At most 100.

  Exemple d'appel

  {"name": "generate_robots", "arguments": {"date": "texte", "agents": {}, "default": "allow", "sitemap": "texte", "sitemaps": [], "categories": {}}}

  Exemple de retour

  {"content": [{"type": "text", "text": "…"}], "structuredContent": {"notes": [], "summary": [], "robots_txt": "texte", "personal_agents_note": "texte"}}

• list_crawlers≈ 280 tokens

  Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.

  Paramètres, retour & exemple (1)

  Paramètres

  • category · string · optionnel · training | dataset | search | user_fetch | other
    Optional: only crawlers in this category.

  Exemple d'appel

  {"name": "list_crawlers", "arguments": {"category": "training"}}

  Exemple de retour

  {"content": [{"type": "text", "text": "…"}], "structuredContent": {"crawlers": [], "generated": "texte", "categories": {}, "unverified": [], "schema_version": 1}}

• validate_robots≈ 359 tokens

  Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.

  Paramètres, retour & exemple (2)

  Paramètres

  • path · string · optionnel
    URL path to test, starting with "/" (default "/").
  • robots_txt · string · requis
    The robots.txt file content.

  Exemple d'appel

  {"name": "validate_robots", "arguments": {"robots_txt": "texte"}}

  Exemple de retour

  {"content": [{"type": "text", "text": "…"}], "structuredContent": {"info": [], "path": "texte", "stats": {}, "valid": true, "errors": [], "warnings": [], "ai_crawlers": [], "diagnostics": {}, "personal_agents": {}}}

C'est votre serveur ?

Revendiquez-le sur l'espace créateur (preuve GitHub, domaine ou npm) pour afficher le badge « propriétaire vérifié » et lancer un pentest comportemental à la demande afin d'obtenir le badge « Vérifié ».

Créer un compte / revendiquer →

Source

https://robotsgate.mike-tusa.workers.dev/mcp