Check robots.txt crawl rules
x402Grade BLast checked 2 hours ago
POST https://web.openverbs.com/v1/robots
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.
Price$0.006per page fetched · paid in USD on Base
Uptime100%
4/4 probes · 270.5 ms
Score77Grade B
Paid usage, 30 days3payers · 3 calls
How to buy
This endpoint takes pay-per-call payments over x402. Call it once without paying to see its live price; any x402 client then pays and retries automatically.
curl -i -X POST "https://web.openverbs.com/v1/robots"
# HTTP/1.1 402 Payment Required: price, asset and payTo are in the responseRecent checks
| Checked (UTC) | Result | Status | Time | Live price |
|---|---|---|---|---|
| 2026-10-11 08:30 | Valid 402 | 402 | 172 ms | $0.006 |
| 2026-10-11 04:15 | Valid 402 | 402 | 320 ms | $0.006 |
| 2026-10-10 23:45 | Valid 402 | 402 | 402 ms | $0.006 |
| 2026-10-10 18:30 | Valid 402 | 402 | 221 ms | $0.006 |
Receipts
No test purchases yet. Graded test purchases are next on our list; until then every number here comes from unpaid checks.
What a check proves