For agents: JSON · MCP

crawl-allowed-check

x402Grade CLast checked 35 min ago

POST https://intel.rallylive.ca/monitor/robots-allowed

Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, the group used, crawl-delay and declared sitemaps. Pre-flight for crawlers and SEO checks. $0.01 per check.

Price$0.01per page fetched · paid in USD on Base

Uptime100%

4/4 probes · 601.5 ms

Score59Grade C

Paid usage, 30 days2payers · 3 calls

Warnings

Wallet mismatch The payout wallet in its live payment answer differs from its catalogue listing on the same network.

How to buy

This endpoint takes pay-per-call payments over x402. Call it once without paying to see its live price; any x402 client then pays and retries automatically.

curl -i -X POST "https://intel.rallylive.ca/monitor/robots-allowed"
# HTTP/1.1 402 Payment Required: price, asset and payTo are in the response

Recent checks

Checked (UTC)	Result	Status	Time	Live price
2026-10-11 10:16	Valid 402	402	548 ms	$0.01
2026-10-11 06:00	Valid 402	402	170 ms	$0.01
2026-10-11 01:30	Valid 402	402	705 ms	$0.01
2026-10-10 21:00	Valid 402	402	655 ms	$0.01

Receipts

No test purchases yet. Graded test purchases are next on our list; until then every number here comes from unpaid checks.

What a check proves