{"endpoint":{"id":"ec030a12c34267d9","rail":"x402","source":"ultravioletadao","url":"https://x402-markdown.onrender.com/v1/crawl/permission","method":"GET","host":"x402-markdown.onrender.com","serviceName":null,"description":"Before fetching, quoting or training on a web page, check the site's machine-readable AI/crawl signals for your crawler. GET /v1/crawl/permission?url=<public http(s) URL>&ua=<robots.txt product token, e.g. GPTBot, ClaudeBot, CCBot, Google-Extended; default *>. Evaluates robots.txt per RFC 9309 (group for ua else *, longest match, Allow wins ties; robots.txt 4xx = no restrictions, 5xx = treat as disallowed), ai.txt, TDMRep (/.well-known/tdmrep.json, tdm-reservation/tdm-policy headers and meta), X-Robots-Tag and <meta name=robots|<ua>> (noai, noimageai, noindex...). Returns verdict 'allowed' | 'disallowed' | 'unclear' plus reasons[] and every raw signal. Disallowed if robots.txt blocks the path for ua or any AI opt-out (noai, tdm-reservation=1, ai.txt disallow) is present; unclear if the page itself could not be checked. Errors are JSON {error:<code>, message, charged:false}. Not charged when robots.txt cannot be retrieved (DNS, connect, timeout) or the URL is not public (400 before paym","tags":[],"platforms":[],"tasks":["markdown","crawl","robots"],"categoryId":"web-extraction","priceUsd":0.005,"priceScheme":"exact","network":"eip155:8453","payTo":"0x64eB63904Cf9688AC5B3087A72AcF9C31a273169","testnetOnly":false,"calls30d":0,"payers30d":0,"lastCalledAt":null,"startFeeUsd":null,"platformUsageExtra":null,"reportedSuccess30d":null,"pageUrl":null,"contributor":null,"batchMax":null,"exampleCount":null,"unitsPerCharge":1,"unitBasis":"call","unitPriceUsd":0.005,"firstSeenAt":"2026-10-29T12:07:18.144Z","lastSeenAt":"2026-10-29T12:48:27.412Z","lastProbedAt":"2026-10-29T12:15:05.351Z","embeddedHash":"4b2d87182fabf4741567087e125eaf2cff795fd0467696670023791d6a7ea608"},"score":{"endpointId":"ec030a12c34267d9","computedAt":"2026-10-29T14:46:00.040Z","score":null,"grade":null,"uptime7d":0,"latencyP50Ms":null,"probeCount7d":1,"flags":["no-usage-data"]},"recentProbes":[{"id":462885,"endpointId":"ec030a12c34267d9","checkedAt":"2026-10-29T12:15:05.504Z","statusCode":503,"latencyMs":492,"valid402":false,"priceUsd":null,"payTo":null,"error":null,"mcpOutcome":null,"mcpProtocol":null}],"siblings":{"total":5,"sameCategory":5,"rows":[{"id":"e36677b669dcebd9","method":"GET","url":"https://x402-markdown.onrender.com/markdown","categoryId":"web-extraction","priceUsd":0.1,"priceScheme":"exact"},{"id":"389f9a16149f44d6","method":"POST","url":"https://x402-markdown.onrender.com/markdown","categoryId":"web-extraction","priceUsd":0.1,"priceScheme":"exact"},{"id":"8b90fb5245c5ff8e","method":"GET","url":"https://x402-markdown.onrender.com/v1/read","categoryId":"web-extraction","priceUsd":0.005,"priceScheme":"exact"},{"id":"a1da35133223d2a4","method":"GET","url":"https://x402-markdown.onrender.com/v1/read?format=chunks","categoryId":"web-extraction","priceUsd":0.02,"priceScheme":"exact"},{"id":"8c51675ae4683596","method":"GET","url":"https://x402-markdown.onrender.com/v1/url/meta","categoryId":"web-extraction","priceUsd":0.005,"priceScheme":"exact"}]},"tools":null,"source":{"name":"Hagglyai","url":"https://hagglyai.com/compare/web-extraction","asOf":"2026-10-29T14:53:57.954Z"}}