For AI agents: a documentation index is available at /llms.txt
Skip to main content

/crawl

POST 

/crawl

Start an asynchronous crawl job that spiders a website and scrapes every discovered page. Returns a crawl ID and status URL for polling results via GET /crawl/{id}. Supports depth control, path filtering, sitemap strategies, and webhook notifications.

Request​

Query Parameters

    profile string
    token string

Body

    urlstringrequired

    The URL to crawl. Must be a valid http or https URL.

    limitnumber

    Maximum number of pages to crawl. Clamped to your plan's limit.

    Possible values: >= 1

    Default value: 100
    maxDepthnumber

    Maximum link-follow depth from the root URL.

    Possible values: >= 0

    Default value: 5
    maxRetriesnumber

    Number of retry attempts per failed page.

    Possible values: >= 0

    Default value: 1
    allowExternalLinksboolean

    Whether to follow links to external domains.

    Default value: false
    allowSubdomainsboolean

    Whether to follow links to subdomains of the root URL.

    Default value: false
    sitemapstring

    Sitemap handling strategy.

    Possible values: [auto, force, skip]

    Default value: auto
    includePathsstring[]

    Regex patterns for URL paths to include.

    Default value: []
    excludePathsstring[]

    Regex patterns for URL paths to exclude.

    Default value: []
    delaynumber

    Delay between requests in milliseconds.

    Possible values: >= 0

    Default value: 200
    scrapeOptions object

    Options controlling how each page is scraped.

    formatsstring[]

    Output formats for scraped content.

    Possible values: [html, markdown, rawText]

    Default value: ["markdown"]
    onlyMainContentboolean

    Whether to extract only the main content of the page.

    Default value: true
    includeTagsstring[]

    HTML tag selectors to include.

    Default value: []
    excludeTagsstring[]

    HTML tag selectors to exclude.

    Default value: []
    waitFornumber

    Time in ms to wait after page load before scraping.

    Possible values: >= 0

    Default value: 0
    headers object

    Custom HTTP headers to send with each request.

    property name*string
    timeoutnumber

    Navigation timeout in milliseconds.

    Possible values: >= 1000 and <= 180000

    Default value: 150000
    proxystring

    Proxy network to route page fetches through: "residential" (default) or "datacenter".

    Possible values: [datacenter, residential]

    webhook object

    Webhook configuration for crawl event notifications.

    urlstringrequired

    The HTTPS URL to send webhook events to.

    eventsstring[]

    Which events to send.

    Possible values: [completed, failed, page]

    Default value: ["completed"]

Responses​

Schema
    successbooleanrequired
    idstringrequired
    urlstringrequired
Was this page helpful?