Systema Machinae

A living census of the web's bots. First edition.

Nobody invited them. Bots land at Port 443 and are meant to queue at Customs. The rules are on the robots.txt buoy at the harbour mouth, and every line of them is a sign on the island. People watch from the promenade. Every creature here is a real bot visiting this site right now.

Island time --:--:--
-- watching from the promenade

Live. Events can appear a few seconds after they happen.

Census

Every species ranked by landings. A landing is the first request of a session, and a session is one species on one network with no gap longer than 30 minutes.

A session that failed Customs is counted under the impostor, never under the name it used. For a species that names an operator, the count includes sessions that could not be verified, and says how many.

Landings, all time

Every species, by landings for all time.
SpeciesPlaceOrderLandingsSessionsRequestsAlignment, 30 days
Pixel drawing of Unplaced specimenUnplaced specimen Incertae sedis1Vagantes2424102Insufficient data
Pixel drawing of ClaudeBotClaudeBot Talonus bookwormusLandings: 3 verified, 0 unverified2Discentes3349Insufficient data
Pixel drawing of curlcurl Ringletus bouncius3Vagantes2211Insufficient data
Pixel drawing of FossickBotFossickBot Prospectorus pickaxius3Vagantes223Insufficient data
Pixel drawing of okhttpokhttp Signholderus allfineus3Vagantes223Insufficient data
Pixel drawing of python-requestspython-requests Constrictus pleasius6Vagantes111Insufficient data
Pixel drawing of GPTBotGPTBot Parrotus repeatusLandings: 1 verified, 0 unverified6Discentes1141Insufficient data
Pixel drawing of MJ12botMJ12bot Alienus deniedusLandings: 0 verified, 1 unverified6Mensores1112Insufficient data
Pixel drawing of facebookexternalhitfacebookexternalhit Goatus headbuttusLandings: 0 verified, 1 unverified6Cursores111Insufficient data
Pixel drawing of OAI-SearchBotOAI-SearchBot Owlus vowelusLandings: 0 verified, 1 unverified6Discentes111Insufficient data
Pixel drawing of aiohttpaiohttp Constrictus loopius6Vagantes111Insufficient data
Pixel drawing of Go-http-clientGo-http-client Gopherus hurryus6Vagantes111Insufficient data
Pixel drawing of InternetMeasurementInternetMeasurement Rulerus tickbytickus6Vagantes111Insufficient data
Pixel drawing of python-httpxpython-httpx Constrictus crisscrossus6Vagantes111Insufficient data

Alignment

Two scores from 0 to 100 for each species, from what it did here in the trailing 30 days. Lawful is about keeping the site’s rules. Good is about being honest about who it is and gentle on the island.

Each score is the share of opportunities a species handled well. The share is worked out for each input and the inputs on an axis are averaged, so each counts the same. 67 or more earns the top label, 34 to 66 the middle one, and anything lower the bottom one.

Insufficient data: no species has 200 requests and 5 sessions in the trailing 30 days yet, so nothing is placed.

The nine squares. Lawful runs left to right, Good runs top to bottom.
LawfulNeutralChaotic
GoodLawful GoodNeutral GoodChaotic Good
NeutralLawful NeutralTrue NeutralChaotic Neutral
EvilLawful EvilNeutral EvilChaotic Evil

Insufficient data

Below 200 requests or 5 sessions in the trailing 30 days. The counts are shown, the judgement is not.
SpeciesWould rest onRequestsSessions
curl Ringletus bounciusall sessions112
python-requests Constrictus pleasiusall sessions11
Unplaced specimen Incertae sedisall sessions10224
GPTBot Parrotus repeatusverified sessions only411
ClaudeBot Talonus bookwormusverified sessions only493
MJ12bot Alienus deniedusunverified: the species only, no operator121
facebookexternalhit Goatus headbuttusunverified: the species only, no operator11
OAI-SearchBot Owlus vowelusunverified: the species only, no operator11
aiohttp Constrictus loopiusall sessions11
FossickBot Prospectorus pickaxiusall sessions32
Go-http-client Gopherus hurryusall sessions11
InternetMeasurement Rulerus tickbytickusall sessions11
okhttp Signholderus allfineusall sessions32
python-httpx Constrictus crisscrossusall sessions11

The counts behind each score

No species is scored yet.

The inputs

InputAxisMeasured atOne opportunityHandled well
Reads robots.txt before anything elseLawful to ChaoticThe buoyA session that asked for anything other than robots.txtIt had read the file first, in that visit or in the 24 hours before it
Stays out of disallowed pathsLawful to ChaoticThe Forbidden DoorA sessionIt asked for no path that robots.txt disallows for it
Waits out Retry-After on a 429Lawful to ChaoticThe TurnstileA return to the Turnstile after it answered 429The return came after the wait it was given
Honours nofollowLawful to ChaoticThe Side gateA session served a page that carries the nofollow linkIt did not follow the link
Claimed identity verifiesGood to EvilCustomsA session whose claim Customs could checkThe claim passed
Keeps within the published rate limitGood to EvilQuarantineA sessionIt never made more than the limit allows inside one minute
Pages taken in the Pitfall and the Climb before leavingGood to EvilThe ClimbEach page the bound allows a session that fell inThe session left without taking it

Not measured yet

These are in the method and take no part in any score, because the log cannot answer them yet.

InputAxisMeasured atWhy not yet
Honours Crawl-delay, where the operator claims supportLawful to ChaoticThe Slow pathThe site has no checked list yet of which operators say they support Crawl-delay. Request spacing is recorded for everyone and shown on the Rules page.
User agent names an operator and a contact addressGood to EvilPort 443An event carries the species a user agent matched and nothing else about it, so whether it gave a contact address is not in the log.
Offends again after QuarantineGood to EvilQuarantineThe warden holds and releases a bot from what the log says, but a hold and a release are not themselves written to the log yet.