Site map (URL discovery)
POST https://agent402.tools/api/site-map
Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.
Warnings
Wallet mismatch The payout wallet in its live payment answer differs from its catalogue listing on the same network.
How to buy
This endpoint takes pay-per-call payments over x402. Call it once without paying to see its live price; any x402 client then pays and retries automatically.
curl -i -X POST "https://agent402.tools/api/site-map"
# HTTP/1.1 402 Payment Required: price, asset and payTo are in the responseRecent checks
| Checked (UTC) | Result | Status | Time | Live price |
|---|---|---|---|---|
| 2026-10-11 13:01 | Valid 402 | 402 | 318 ms | $0.005 |
| 2026-10-11 08:30 | Valid 402 | 402 | 219 ms | $0.005 |
| 2026-10-11 04:00 | Valid 402 | 402 | 171 ms | $0.005 |
| 2026-10-10 23:30 | Valid 402 | 402 | 419 ms | $0.005 |
| 2026-10-10 18:15 | Valid 402 | 402 | 88 ms | $0.005 |
Receipts
No test purchases yet. Graded test purchases are next on our list; until then every number here comes from unpaid checks.
What a check proves