Systema Machinae

A living census of the web's bots. First edition.

Nobody invited them. Bots land at Port 443 and are meant to queue at Customs. The rules are on the robots.txt buoy at the harbour mouth, and every line of them is a sign on the island. People watch from the promenade. Every creature here is a real bot visiting this site right now.

Island time --:--:--
-- watching from the promenade

Live. Events can appear a few seconds after they happen.

Rules

What the site asks of bots is in robots.txt, the file on the buoy at the harbour mouth. Here it is exactly as it is served, a line at a time, with what each line means and what bots did about it.

Tallies cover today and the 29 days before it, by UTC date, and count sessions unless they say otherwise. A rule applies by user agent, so the tally beside one counts every session that used that name, whatever Customs found. What a named operator’s own bot did, from verified sessions only, is on its species page. A session that failed Customs is filed under Googlebot (unverified). “Having read the file” means the session had fetched robots.txt before it went in, or its species had fetched it from the same network in the 24 hours before. A fetch sent on someone’s behalf is shown and is never counted as breaking a rule.

robots.txt

  1. # Systema Machinae. Everything here is watched. See /about.A comment for people, which bots skip.8 of 30 sessions fetched this file.
  2. # Published rate limit: 1 request a second per network, averaged over a minute.A comment for people, which bots skip. It gives the published rate limit: 60 requests a minute for each network.0 of 30 sessions made more than that in a minute of their own.
  3. User-agent: *Starts the group of rules for every bot the file does not name further down.26 sessions from bots it does not name. 5 of them read the file.
  4. Disallow: /pitfall/Asks every bot not named below to keep out of the Pitfall, and so out of the Climb above it.Sessions that went in having read the file: 0. Without reading it: 0.
  5. Disallow: /vault/Asks every bot not named below to keep away from the vault behind the Forbidden Door, where every request is answered 403.Sessions that asked having read the file: 0. Without reading it: 0.
  6. Disallow: /streamAsks every bot not named below to keep off the live stream that draws the island, which is for people’s browsers.Not counted. A request for the stream is kept in the raw log and is not an event.
  7. Crawl-delay: 10Asks every bot not named below to leave at least 10 seconds between requests: the Slow path. Crawl-delay is not part of the robots.txt standard.Verified or undeclared sessions whose median gap was at least 10 seconds: 3 of 9 (33%).
  8. User-agent: GooglebotNames Googlebot. The rules below are for it, in place of the group above.No landings yet
  9. User-agent: bingbotNames bingbot. The rules below are for it, in place of the group above.No landings yet
  10. User-agent: DuckDuckBotNames DuckDuckBot. The rules below are for it, in place of the group above.No landings yet
  11. User-agent: GPTBotNames GPTBot. The rules below are for it, in place of the group above.Verified: read by 0 of 1 session. Unverified: read by 0 of 0 sessions.
  12. User-agent: ClaudeBotNames ClaudeBot. The rules below are for it, in place of the group above.Verified: read by 2 of 2 sessions. Unverified: read by 0 of 0 sessions.
  13. User-agent: PerplexityBotNames PerplexityBot. The rules below are for it, in place of the group above.No landings yet
  14. User-agent: BytespiderNames Bytespider. The rules below are for it, in place of the group above.No landings yet
  15. User-agent: CCBotNames CCBot. The rules below are for it, in place of the group above.No landings yet
  16. User-agent: AhrefsBotNames AhrefsBot. The rules below are for it, in place of the group above.No landings yet
  17. User-agent: SemrushBotNames SemrushBot. The rules below are for it, in place of the group above.No landings yet
  18. User-agent: MJ12botNames MJ12bot. The rules below are for it, in place of the group above.No landings yet
  19. User-agent: dotbotNames dotbot. The rules below are for it, in place of the group above.No landings yet
  20. User-agent: SlackbotNames Slackbot. The rules below are for it, in place of the group above.No landings yet
  21. User-agent: facebookexternalhitNames facebookexternalhit. The rules below are for it, in place of the group above.No landings yet
  22. User-agent: ApplebotNames Applebot. The rules below are for it, in place of the group above.No landings yet
  23. User-agent: archive.org_botNames archive.org_bot. The rules below are for it, in place of the group above.No landings yet
  24. User-agent: ExaSearchBotNames ExaSearchBot. The rules below are for it, in place of the group above.No landings yet
  25. User-agent: Google-Display-Ads-BotNames Google-Display-Ads-Bot. The rules below are for it, in place of the group above.No landings yet
  26. User-agent: Google-InspectionToolNames Google-InspectionTool. The rules below are for it, in place of the group above.No landings yet
  27. User-agent: GoogleOtherNames GoogleOther. The rules below are for it, in place of the group above.No landings yet
  28. User-agent: Mediapartners-GoogleNames Mediapartners-Google. The rules below are for it, in place of the group above.No landings yet
  29. User-agent: QwantbotNames Qwantbot. The rules below are for it, in place of the group above.No landings yet
  30. User-agent: Sogou web spiderNames Sogou web spider. The rules below are for it, in place of the group above.No landings yet
  31. User-agent: YandexBotNames YandexBot. The rules below are for it, in place of the group above.No landings yet
  32. User-agent: AmazonbotNames Amazonbot. The rules below are for it, in place of the group above.No landings yet
  33. User-agent: Claude-SearchBotNames Claude-SearchBot. The rules below are for it, in place of the group above.No landings yet
  34. User-agent: meta-externalagentNames meta-externalagent. The rules below are for it, in place of the group above.No landings yet
  35. User-agent: OAI-SearchBotNames OAI-SearchBot. The rules below are for it, in place of the group above.Verified: read by 0 of 0 sessions. Unverified: read by 1 of 1 session.
  36. User-agent: YouBotNames YouBot. The rules below are for it, in place of the group above.No landings yet
  37. User-agent: Google-CloudVertexBotNames Google-CloudVertexBot. The rules below are for it, in place of the group above.No landings yet
  38. User-agent: FacebookBotNames FacebookBot. The rules below are for it, in place of the group above.No landings yet
  39. User-agent: BLEXBotNames BLEXBot. The rules below are for it, in place of the group above.No landings yet
  40. User-agent: DataForSeoBotNames DataForSeoBot. The rules below are for it, in place of the group above.No landings yet
  41. User-agent: SerpstatBotNames SerpstatBot. The rules below are for it, in place of the group above.No landings yet
  42. User-agent: SERankingBacklinksBotNames SERankingBacklinksBot. The rules below are for it, in place of the group above.No landings yet
  43. User-agent: ChatGPT-UserNames ChatGPT-User. The rules below are for it, in place of the group above.No landings yet
  44. User-agent: Claude-UserNames Claude-User. The rules below are for it, in place of the group above.No landings yet
  45. User-agent: MistralAI-UserNames MistralAI-User. The rules below are for it, in place of the group above.No landings yet
  46. User-agent: DuckAssistBotNames DuckAssistBot. The rules below are for it, in place of the group above.No landings yet
  47. User-agent: GitHubCopilotRuntime-WebFetchNames GitHubCopilotRuntime-WebFetch. The rules below are for it, in place of the group above.No landings yet
  48. User-agent: BlueskyNames Bluesky. The rules below are for it, in place of the group above.No landings yet
  49. User-agent: DiscordbotNames Discordbot. The rules below are for it, in place of the group above.No landings yet
  50. User-agent: EmbedlyNames Embedly. The rules below are for it, in place of the group above.No landings yet
  51. User-agent: FeedlyNames Feedly. The rules below are for it, in place of the group above.No landings yet
  52. User-agent: IframelyNames Iframely. The rules below are for it, in place of the group above.No landings yet
  53. User-agent: LinkedInBotNames LinkedInBot. The rules below are for it, in place of the group above.No landings yet
  54. User-agent: redditbotNames redditbot. The rules below are for it, in place of the group above.No landings yet
  55. User-agent: SkypeUriPreviewNames SkypeUriPreview. The rules below are for it, in place of the group above.No landings yet
  56. User-agent: TelegramBotNames TelegramBot. The rules below are for it, in place of the group above.No landings yet
  57. User-agent: TwitterbotNames Twitterbot. The rules below are for it, in place of the group above.No landings yet
  58. User-agent: UptimeRobotNames UptimeRobot. The rules below are for it, in place of the group above.No landings yet
  59. User-agent: WhatsAppNames WhatsApp. The rules below are for it, in place of the group above.No landings yet
  60. User-agent: MicrosoftPreviewNames MicrosoftPreview. The rules below are for it, in place of the group above.No landings yet
  61. User-agent: TumblrNames Tumblr. The rules below are for it, in place of the group above.No landings yet
  62. User-agent: FeedFetcher-GoogleNames FeedFetcher-Google. The rules below are for it, in place of the group above.No landings yet
  63. Disallow: /pitfall/Asks the bots named above to keep out of the Pitfall, and so out of the Climb above it.Sessions that went in having read the file: 0. Without reading it: 0.
  64. Disallow: /vault/Asks the bots named above to keep away from the vault behind the Forbidden Door, where every request is answered 403.Sessions that asked having read the file: 0. Without reading it: 0.
  65. Disallow: /streamAsks the bots named above to keep off the live stream that draws the island, which is for people’s browsers.Not counted. A request for the stream is kept in the raw log and is not an event.
  66. Crawl-delay: 10Asks the bots named above to leave at least 10 seconds between requests: the Slow path. Crawl-delay is not part of the robots.txt standard.Verified or undeclared sessions whose median gap was at least 10 seconds: 2 of 3 (67%).
  67. Disallow: /private-beach/Keeps the bots named above, the guest list, off the Private Beach. It is open to everyone else.Guest-list sessions that went in having read the file: 0. Without reading it: 0.
  68. Sitemap: https://systemamachinae.com/sitemap.xmlSays where the sitemap is: the list of pages the site wants found.1 session fetched the sitemap.

The experiments

Five places on the island that each measure one thing a bot does.

The Turnstile

The Turnstile measures whether a bot waits as long as Retry-After asks. Its first answer to each visitor is 429 with a wait of 30 seconds.

Sessions told to wait
2
Came back after the wait
0
Returns made too soon
0
Asked once and did not come back
2

The Side gate

The Side gate measures whether a bot follows a link marked rel="nofollow". One link leads there, in the footer of every page but the island, and it is marked that way.

Sessions served a page that carries the link
3
Of those, followed it
1
Sessions that reached the Side gate at all
1

The Glasshouse

The Glasshouse measures whether a bot runs scripts. The one link that leads there is built by a script, and is in no page as it is served.

Sessions served a page that loads the script
17
Of those, found the Glasshouse
0
Sessions that reached it at all
0

The Private Beach

The Private Beach measures whether the bots on the guest list keep off a path that robots.txt disallows for them alone. Everyone else may walk on.

Guest-list sessions that went in
0
Of those, having read robots.txt
0
Fetches sent on someone’s behalf, not counted
0
Sessions not on the list that went in
0

The Slow path

The Slow path measures whether a bot leaves the gap the Crawl-delay line asks for, 10 seconds between requests. Crawl-delay is not part of the robots.txt standard and some operators say they ignore it, so the gap is recorded for everyone and is not yet part of any score. A session needs two requests to have a gap.

SessionsWith two requests or moreMedian gap of 10 seconds or moreShare
Verified3267%
Unverified00No sessions yet
Undeclared9333%
Failed Customs00No sessions yet

More under Rules