For AI agents: a documentation index is available at /llms.txt
Skip to main content

/smart-scrape

POST 

/smart-scrape

Intelligently scrapes a URL using cascading strategies (HTTP fetch, proxy, headless browser, etc).

Request​

Query Parameters

    deviceType string

    Possible values: [phone, tablet]

    emulatedDevice string
    emulationOs Platform

    Possible values: [android, ios, linux, macos, windows]

    profile string
    timeout number
    token string
    allowedDomains string[]
    credentials object
    externalProxyServer string
    integrationId string
    proxy ProxyType

    Possible values: [datacenter, residential]

    proxyCity string
    proxyCountry string
    proxyLocaleMatch string

    Possible values: [0, 1, false, true]

    proxyPreset string
    proxyState string
    proxySticky string

    Possible values: [0, 1, false, true]

Body

    urlstringrequired

    The URL to scrape. Must be an http or https URL.

    formatsstring[]

    Output formats to include in the response. Accepts an array of format strings such as ["markdown", "screenshot"]. When screenshot or pdf is included, a browser strategy is forced. Mirrors the Firecrawl "formats" convention.

    • markdown – page content converted to markdown
    • html – cleaned HTML (returned by default in content, this is a no-op convenience value)
    • rawText – DOM text with script, style, and noscript elements removed and whitespace collapsed; PDF targets return extracted text
    • screenshot – base64 PNG by default, JPEG when screenshotOptions.type is jpeg (forces browser strategy)
    • pdf – PDF of the page as base64-encoded string (forces browser strategy)
    • links – list of links extracted from the page
    • json – AI-extracted object per jsonOptions, without forcing a browser
    • summary – short AI-written summary, without forcing a browser
    • changeTracking – compare with the last tracked check; requires markdown and always scrapes fresh

    Possible values: [changeTracking, html, json, links, markdown, pdf, rawText, screenshot, summary]

    Default value: ["html"]
    changeTrackingOptions object

    Options for changeTracking; requires that format and explicit markdown. History is private to the account, URL and tag for 30 days after the last tracked success. Keep content-shaping options consistent between checks; proxy and emulationOs do not select separate histories.

    modesstring[]

    Only git-diff is supported; other modes are rejected by the handler.

    tagstring | nullnullable

    Keeps separate history for the same URL.

    jsonOptions object

    Inputs for the json format; requires json in formats and at least a schema or prompt.

    schema object

    Best-effort JSON Schema; at most 20,000 characters when serialized.

    object
    property name*any

    Best-effort JSON Schema; at most 20,000 characters when serialized.

    promptstring

    Instruction guiding the extraction.

    Possible values: <= 10000 characters

    actions[]

    Browser actions to run in order before the final page is captured. The supported action types are wait, click, write, scroll, scrape, and executeJavascript. A maximum of 20 actions is allowed. Detailed, index-aware validation is performed by the route so the generated schema intentionally leaves each array entry permissive.

    • wait accepts exactly one of milliseconds or selector
    • click accepts a non-empty selector
    • write accepts a non-empty selector and string text
    • scroll optionally accepts a selector
    • scrape captures the current page URL and HTML
    • executeJavascript accepts a non-empty script up to 10,000 characters

    wait, click, write, scroll, and executeJavascript may include a positive timeout of at most 30,000 milliseconds (default 10,000). Every action and the final page settle share the request timeout budget.

    An empty array is accepted as a no-op. Use a trailing newline in write.text to submit a form. Screenshots and PDFs are requested through top-level formats and capture the final page after all actions run.

    Possible values: <= 20

    proxystring

    The proxy network to route the scrape through. Defaults to residential. Use datacenter for the cheaper datacenter pool.

    Possible values: [datacenter, residential]

    location object

    Geo-targeting. country is a case-insensitive ISO 3166-1 alpha-2 code. Forces proxy egress from that country, skipping direct HTTP fetch, and selects its browser time zone where the existing country mapping supports it. Omitted preserves today's behavior.

    countrystring
    blockAdsboolean

    Block ads and trackers in browser tiers.

    Default value: false
    screenshotOptions object

    Screenshot tuning; requires "screenshot" in formats.

    fullPageboolean

    Whole page (default) or only the viewport.

    Default value: true
    typestring

    Image format.

    Possible values: [jpeg, png]

    Default value: png
    qualitynumber

    JPEG quality, integer 1-100. Only valid with type "jpeg".

    viewport object

    Viewport in CSS pixels, integers 1-4096. Not combinable with emulationOs.

    widthnumberrequired
    heightnumberrequired
    onlyMainContentboolean

    For HTML webpages, extract the main article content, removing navigation, headers, footers, sidebars and other boilerplate from DOM-derived outputs. Parsed JSON and PDF content are unchanged. Defaults to true, like Firecrawl and /crawl, except when includeTags is sent; send false for the full page.

    includeTagsstring[]

    CSS selectors to keep in HTML webpage outputs. Malformed entries are ignored; if no selector matches, unfiltered content is returned. Mutually exclusive with excludeTags and onlyMainContent.

    Possible values: <= 100

    excludeTagsstring[]

    CSS selectors to remove from HTML webpage outputs. Malformed entries are ignored. Mutually exclusive with includeTags.

    Possible values: <= 100

    headers object

    Custom request headers sent to the target. host, authorization, proxy-authorization, cookie, set-cookie, x-forwarded-for, x-real-ip, and forwarded are removed.

    property name*string
    waitFornumber

    Milliseconds to wait after page load. Clamped to [0, 30000]; positive values force a browser strategy.

    Default value: 0
    strategyCacheboolean

    Whether to use and update the remembered winning strategy for this host. Defaults to true. Set to false to bypass both reads and writes for this request.

    Default value: true
    maxAgeinteger

    Return a cached result for this account if one younger than this many milliseconds exists for the same url and output options, and cache this request's successful result for up to the same period (at most 48 hours). 0 (default) disables caching. Must be a non-negative integer. Ignored for changeTracking requests and for requests that use profile, non-empty actions, or custom headers that remain after sanitizing (headers that are always stripped do not count).

    Possible values: >= 0

    Default value: 0
    storeInCacheboolean

    Whether a successful result of a request with maxAge above 0 may be cached for later requests. Set to false to read the cache without adding to it.

    Default value: true

Responses​

An HTML payload of the scraped page content.

Schema
    okbooleanrequired

    Whether the scrape was successful or not.

    statusCodenumberrequired

    The HTTP status code of the response, or null if the scrape failed before an HTTP response was received (e.g. due to a network error or captcha).

    content objectrequired

    The content of the scraped page. This is extracted text for PDF targets, a string for HTML content, or a parsed object for JSON. Will be null if the scrape was unsuccessful, or if only screenshot/PDF captures were requested and included content exceeded the serialization-work budget.

    anyOf
    object
    contentTypestringrequired

    The content type returned by the target, or null if the scrape was unsuccessful or unavailable. JSON content is parsed into an object; PDF content is extracted into text.

    headersobjectrequired

    The HTTP response headers returned by the site, or an empty object if unavailable.

    strategystringrequired

    The strategy that ultimately succeeded in scraping the page, or the strategy that was being attempted when the scrape failed.

    attemptedstring[]required

    The strategies that were attempted during the scrape, in order.

    messagestringrequired

    An error message describing why the scrape failed, or null if the scrape was successful.

    screenshotstringrequired

    A base64 PNG by default, JPEG when screenshotOptions.type is jpeg, present when "screenshot" is in formats. On a successful scrape this is the scraped page; on a captcha / anti-bot failure (ok:false) it is a best-effort debug capture of the blocking page. Null otherwise.

    pdfstringrequired

    A base64-encoded PDF of the page, present when "pdf" is in formats and the scrape succeeded.

    markdownstringrequired

    The page content converted to markdown, present when "markdown" is in formats and the scrape succeeded. For PDF targets, this is extracted page text joined with blank lines.

    linksstring[]required

    A list of links found on the page, present when "links" is in formats and the scrape succeeded.

    actions objectnullablerequired

    Values captured by scrape and executeJavascript actions, or null when actions were omitted, empty, or the scrape failed.

    scrapes object[]required
  • Array [
  • urlstringrequired
    htmlstringrequired
  • ]
  • javascriptReturns object[]required
  • Array [
  • typestringrequired
    valuestringnullablerequired
  • ]
  • rawTextstringrequired

    DOM text with scripts/styles removed and whitespace collapsed, or extracted PDF text.

    metadata objectrequired

    Page metadata on success, or null when scraping fails.

    titlestringrequired
    descriptionstringrequired
    languagestringrequired
    sourceURLstringrequired
    statusCodenumberrequired
    cacheStatestring

    Present only when caching was considered: hit or miss.

    Possible values: [hit, miss]

    cachedAtstring

    ISO-8601 capture time of the cached result; present only on hits.

    json objectrequired
    anyOf

    Object extracted per jsonOptions on success; null otherwise.

    object
    property name*any

    Object extracted per jsonOptions on success; null otherwise.

    summarystring | nullnullablerequired

    AI-written summary when requested and successful; null otherwise.

    warningstring | nullnullablerequired

    Why requested extraction is missing or partial on a successful scrape; null otherwise.

    changeTracking objectrequired
    anyOf

    Null when not requested, history is unavailable, markdown is missing, or the scrape failed other than 404/410. previousScrapeAt is the last successful tracked check, not the last change. A status is never guessed.

    changeStatusstringrequired

    Possible values: [changed, new, removed, same]

    previousScrapeAtstring | nullnullablerequired
    visibilitystringrequired
    Constant value: visible
    diff objectrequired
    anyOf

    Only for git-diff on changed pages within size/time limits.

    textstringrequired
    json objectrequired
    files object[]required
  • Array [
  • fromstring | nullnullablerequired
    tostring | nullnullablerequired
    chunks object[]required
  • Array [
  • contentstringrequired
    changes object[]required
  • Array [
  • typestringrequired

    Possible values: [add, del, normal]

    contentstringrequired
    lnnumber
    ln1number
    ln2number
    addboolean
    delboolean
    normalboolean
  • ]
  • ]
  • ]
Was this page helpful?