NetIntel
x402Grade BLast checked 3 hours ago
GET https://netintel.dev/robots-txt/analyze
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Price$0.01per page fetched · paid in USD on Base
Uptime100%
4/4 probes · 400 ms
Score77Grade B
Paid usage, 30 days3payers · 4 calls
How to buy
This endpoint takes pay-per-call payments over x402. Call it once without paying to see its live price; any x402 client then pays and retries automatically.
curl -i -X GET "https://netintel.dev/robots-txt/analyze"
# HTTP/1.1 402 Payment Required: price, asset and payTo are in the responseRecent checks
| Checked (UTC) | Result | Status | Time | Live price |
|---|---|---|---|---|
| 2026-10-11 08:15 | Valid 402 | 402 | 267 ms | $0.01 |
| 2026-10-11 03:45 | Valid 402 | 402 | 533 ms | $0.01 |
| 2026-10-10 23:30 | Valid 402 | 402 | 757 ms | $0.01 |
| 2026-10-10 18:00 | Valid 402 | 402 | 165 ms | $0.01 |
Receipts
No test purchases yet. Graded test purchases are next on our list; until then every number here comes from unpaid checks.
What a check proves