sitemapTurn a public site map into capture-ready URLs.

/sitemap discovers public pages from sitemaps, robots hints, known pages, and optional link crawl settings. Use it before RAG ingestion, bulk capture, competitive research, and public site audits.

Get your free API key

50 MB free. No credit card.

curl -X POST https://api.bytekit.com/v1/sitemap \
  -H "Authorization: Bearer $BYTEKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://stg.bytekit.com",
    "strategy": "standard",
    "max_urls": 5000,
    "webhook_url": "https://api.acme.com/bytekit-hook"
  }'

Response

{ }

No response yet

Click Run to send the request

when to use it

RAG source discovery

Find docs, blogs, changelogs, and help center pages.

Public site audits

Inventory public pages before capture, QA, or migration work.

Competitor maps

See what pages exist before you decide what to monitor.

strategies

Strategy	Use it when	Tradeoff
`quick`	You need known URLs.	Less discovery depth.
`standard`	You need coverage.	Good default.
`deep`	You need more depth.	More bandwidth and time.

output

Gzipped CSV

Download discovered URLs into your ingestion job, warehouse, or review flow.

Job metadata

URL count, source counts, byte usage, warnings, timestamps, status.

Cache-aware repeats

Repeated domain requests can be served faster and cheaper when freshness allows.

Bulk-ready next step

Feed results into /bulk or pick pages for /monitors.

what it is not

Sitemap crawl is not Common Crawl. It is not a promise to discover every hidden route on the internet. It is a practical way to collect public URLs for a site you want to ingest or monitor.

Find the pages before you capture them.

Get your free API key Read the sitemap docs