Crawl Pre-Flight
GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
Listed in Coinbase x402 Bazaar
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
How to buy
This endpoint takes pay-per-call payments over x402. Call it once without paying to see its live price; any x402 client then pays and retries automatically.
curl -i -X GET "https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain"
# HTTP/1.1 402 Payment Required: price, asset and payTo are in the responseEstimated calls per day
Paid calls to crawl-preflight.postnov01.workers.dev's tracked endpoints each day, estimated from the 30-day totals the x402 catalogues report. Each bar is the change in that 30-day total since the day before, plus the calls that left the 30-day window (taken as an average day). It is an estimate: Hagglyai does not measure or pay for these calls.
Usage available for 1 of 1 tracked endpoint, 1 paid calls in the last 30 days. Last updated 3 hours ago. Unknown usage is left out, not counted as zero.
Compared with other providers
crawl-preflight.postnov01.workers.dev ranks 228th of 247 in Web page extraction by its latest reported 30-day paid calls. Coverage matters: a provider whose endpoints report more of their usage can rank higher.
| # | Provider | Reported 30-day calls | Coverage |
|---|---|---|---|
| 1 | stableenrich.dev | 28,177 | 15 of 31 endpoints |
| 2 | win.oneshotagent.com | 15,591 | 13 of 36 endpoints |
| 3 | x402.orthogonal.com | 4,075 | 189 of 249 endpoints |
| 4 | agent402.tools | 3,120 | 142 of 217 endpoints |
| 5 | agent.pocket.network | 2,931 | 13 of 24 endpoints |
| 6 | api.losbeto.xyz | 1,297 | 20 of 107 endpoints |
| 7 | x402.ottoai.services | 1,258 | 28 of 32 endpoints |
| 8 | api.x402node.dev | 1,247 | 162 of 194 endpoints |
| 228 | crawl-preflight.postnov01.workers.dev (this one) | 1 | 1 of 1 endpoint |