/crawl
POST/crawl
Start an asynchronous crawl job that spiders a website and scrapes every discovered page. Returns a crawl ID and status URL for polling results via GET /crawl/{id}. Supports depth control, path filtering, sitemap strategies, and webhook notifications.
Request
Query Parameters
Body
- Schema
- JSON
The URL to crawl. Must be a valid http or https URL.
Maximum number of pages to crawl. Clamped to your plan's limit.
Possible values: >= 1
100Maximum link-follow depth from the root URL.
Possible values: >= 0
5Number of retry attempts per failed page.
Possible values: >= 0
1Whether to follow links to external domains.
falseWhether to follow links to subdomains of the root URL.
falseSitemap handling strategy.
Possible values: [auto, force, skip]
autoRegex patterns for URL paths to include.
[]Regex patterns for URL paths to exclude.
[]Delay between requests in milliseconds.
Possible values: >= 0
200scrapeOptions object
Options controlling how each page is scraped.
Output formats for scraped content.
Possible values: [html, markdown, rawText]
["markdown"]Whether to extract only the main content of the page.
trueHTML tag selectors to include.
[]HTML tag selectors to exclude.
[]Time in ms to wait after page load before scraping.
Possible values: >= 0
0headers object
Custom HTTP headers to send with each request.
Navigation timeout in milliseconds.
Possible values: >= 1000 and <= 180000
150000Proxy network to route page fetches through: "residential" (default)
or "datacenter".
Possible values: [datacenter, residential]
webhook object
Webhook configuration for crawl event notifications.
The HTTPS URL to send webhook events to.
Which events to send.
Possible values: [completed, failed, page]
["completed"]{
"url": "string",
"limit": 100,
"maxDepth": 5,
"maxRetries": 1,
"allowExternalLinks": false,
"allowSubdomains": false,
"sitemap": "auto",
"includePaths": [
"string"
],
"excludePaths": [
"string"
],
"delay": 200,
"scrapeOptions": {
"formats": [
"html"
],
"onlyMainContent": true,
"includeTags": [
"string"
],
"excludeTags": [
"string"
],
"waitFor": 0,
"headers": {},
"timeout": 150000,
"proxy": "datacenter"
},
"webhook": {
"url": "string",
"events": [
"completed"
]
}
}
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
Schema
- Fields
- Example (auto)
{
"success": true,
"id": "string",
"url": "string"
}
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.