API reference

Crawl

Crawl an entire site and return every page in the format you choose.

POSThttps://api.scrapeflow.dev/v1/web/crawl1 credit per page crawled

Starts an async crawl job. Poll the returned `job_id` or set a `webhook` to receive results when the crawl finishes. Runs on SQS-backed workers.

Request

curl -X POST ${SCRAPEFLOW_API_URL}/v1/web/crawl \
  -H "Authorization: Bearer $SCRAPEFLOW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.stripe.com", "limit": 500 }'

Parameters

Parameter	Type	Description
urlrequired	string	The root URL to crawl.
limit	number	Max pages to crawl. Default 100.
includePaths	string[]	Only crawl paths matching these globs.
webhook	string	URL to POST results to when the job completes.

Response

json

{ "success": true, "job_id": "job_a1b2c3", "status": "queued" }

← Answers Map →