For AI agents: a documentation index is available at /llms.txt
Skip to main content

Smart Scrape API

Intelligently scrape any URL using cascading strategies that automatically escalate from fast HTTP fetching to headless browsers and captcha solving as needed. Specify output formats to receive HTML, markdown, screenshots, PDFs, or extracted links, all in a single request.

Endpoint

  • Method: POST
  • Path: /smart-scrape
  • Auth: token query parameter (?token=)
  • Content-Type: application/json
  • Response: application/json
Prerequisites

Quickstart​

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://news.ycombinator.com/",
  "formats": ["html", "markdown", "links"]
}'

Response

{
  "ok": true,
  "statusCode": 200,
  "content": "<html lang=\"en\" op=\"news\"><head><meta name=\"referrer\" content=\"origin\">...</html>",
  "contentType": "text/html; charset=utf-8",
  "headers": {
    "content-type": "text/html; charset=utf-8",
    "cache-control": "private; max-age=0"
  },
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "message": null,
  "actions": null,
  "screenshot": null,
  "pdf": null,
  "markdown": "# Hacker News\n\n[new](newest) | [past](front) | [comments](newcomments) | [ask](ask) | [show](show) | [jobs](jobs) | [submit](submit)\n\n1. [Motorola GrapheneOS devices will be bootloader unlockable/relockable](https://grapheneos.social/...)...",
  "rawText": null,
  "links": [
    "https://news.ycombinator.com/news",
    "https://news.ycombinator.com/newest",
    "https://news.ycombinator.com/front"
  ],
  "metadata": {
    "title": "Hacker News",
    "description": null,
    "language": "en",
    "sourceURL": "https://news.ycombinator.com/",
    "statusCode": 200
  }
}

How it works​

The Smart Scrape API uses a cascading strategy pipeline to fetch content in the most efficient way possible. It starts with the fastest, cheapest approach and automatically escalates to heavier strategies only when needed:

  1. Fast HTTP fetch: Makes a lightweight HTTP request that mimics a real browser's network fingerprint. This handles the majority of static and server-rendered sites in under 2 seconds.

  2. Proxied HTTP fetch: If the initial request is blocked (e.g., by IP detection), the same request is retried through the selected proxy network (residential by default, or datacenter if proxy: "datacenter" is set in the request body).

  3. Headless browser: If the page requires JavaScript rendering (single-page apps, client-rendered content), a full stealth browser is launched to render the page.

  4. Browser + captcha solving: If a captcha or bot detection challenge is encountered, the browser automatically detects and solves it (supports reCAPTCHA, Cloudflare Turnstile, and others).

The pipeline stops as soon as a strategy succeeds. The strategy field in the response tells you which approach was used, and the attempted array shows the full sequence of strategies tried.

Captcha handling scope

Smart Scrape only solves captchas that gate access to the page itself — for example, a Cloudflare Turnstile interstitial or a reCAPTCHA that blocks the page from loading. In these cases the browser solves the challenge automatically so the underlying content can be returned.

Captchas that are embedded in a form on the page (e.g., a reCAPTCHA next to a "Submit" button on a contact or signup form) are not solved. Actions can interact with ordinary form controls, but they do not solve embedded form captchas. If you need to submit a form behind a captcha, use BrowserQL with the solve mutation instead.

Unit cost​

The time price follows the strategy that served or ended the request, returned in the response's strategy field. Request time is rounded up to the next 30-second increment.

StrategyTime cost
Plain fetch without a proxy (http-fetch)0.5 units per 30 seconds
Proxied fetch (http-proxy)1 unit per 30 seconds
Browser (browser)1 unit per 30 seconds
Browser + captcha solving (browser-captcha)1 unit per 30 seconds
Cached result (cache)0.5 units per 30 seconds

A cached result is the cheapest way to serve a request. Hits return in well under 30 seconds, so a hit costs 0.5 units, with no proxy, captcha, or AI extraction charges.

The json and summary formats add AI extraction units on top of the scrape's own cost. They're priced on the model tokens used: 1 unit per 6,400 input tokens plus 1 unit per 800 output tokens, combined and rounded up to a whole unit per request. A typical product page costs 1 extraction unit, and a long page cut at the 100,000-character input limit costs about 5. You're billed for the tokens of every model call that returns a response, even if the result is unusable (for example, json isn't a JSON object or summary comes back empty). Calls that error or time out, and extraction that's skipped, cost no extraction units.

Proxy traffic adds 6 units per MB for residential proxies or 2 units per MB for datacenter proxies. Each successful captcha solve adds 10 units. These charges include proxy traffic and successful captcha solves from every attempted step, including steps that failed before the winning one. For example, if a remembered browser strategy fails and plain fetch then serves the page, its time costs 0.5 units per 30 seconds, but the earlier step's proxy and captcha charges still apply.

Your balance and invoices count whole units. A fractional charge is carried and settled when a later fractional charge completes a whole unit: two simple scrapes of up to 30 seconds each cost one unit. Your balance can trail your true usage by less than one unit. The dashboard Logs page shows each request's exact units, such as 0.5.

Request body​

FieldTypeRequiredDefaultDescription
urlstringYes-The URL to scrape. Must be http:// or https://.
formatsstring[]No["html"]Output formats to include. Options: "html", "markdown", "rawText", "screenshot", "pdf", "links", "json", "summary", "changeTracking". Must contain at least one format. "json" and "summary" use AI extraction and don't force a browser. "changeTracking" compares the page with your last check and requires "markdown".
jsonOptionsobjectNo-Extraction inputs for the "json" format: schema (a JSON Schema object, at most 20,000 characters serialized) and/or prompt (at most 10,000 characters). Required when formats includes "json", and rejected otherwise.
changeTrackingOptionsobjectNo-Options for the "changeTracking" format: modes (only ["git-diff"], which adds a diff of changed lines) and tag (a non-empty string that keeps a separate history for the same URL; null or omitted means no tag). Rejected unless formats includes "changeTracking".
maxAgeintegerNo0Serve a cached result for your account if one is younger than this many milliseconds, and cache this request's result for the same period, up to 48 hours. 0 disables caching.
storeInCachebooleanNotrueWhen maxAge is above 0, set to false to read the cache without saving this request's result.
proxystringNo"residential"Proxy network to route the scrape through: "residential" (6 units/MB) or "datacenter" (2 units/MB).
onlyMainContentbooleanNotrueRemoves nav, footer, aside, [role="navigation"], script, style, and noscript elements. Defaults to true, like Firecrawl, unless you send includeTags. Send false to get the full page.
includeTagsstring[]No[]Keeps HTML webpage content matching up to 100 CSS selectors. Malformed entries are ignored; if no selector matches, unfiltered content is returned. It can't be combined with excludeTags or onlyMainContent: true. Explicit onlyMainContent: false is accepted; omitting it defaults to false.
excludeTagsstring[]No[]Removes HTML webpage content matching up to 100 CSS selectors. Malformed selectors are ignored. It can't be combined with includeTags.
headersobjectNo{}Adds request headers sent to the target. Unsafe transport, proxy, authentication, and cookie headers are removed.
waitFornumberNo0Waits after page load before reading content. Values are clamped to 0–30,000 milliseconds, and any positive value forces a browser strategy.
actionsobject[]No-Sequential browser actions to run before the final page is captured. A non-empty array forces a browser strategy.

Actions​

Use actions to interact with a rendered page before Smart Scrape returns its final content. Actions run in submission order after page-level anti-bot checks have completed. The first failed action aborts the sequence, returns ok: false, and discards partial action results.

TypeFieldsBehavior
waitExactly one of milliseconds or selector; optional timeoutWait for a duration or for a visible selector.
clickselector; optional timeoutClick the first matching element.
writeselector, text; optional timeoutType into an element. A trailing \n sends Enter.
scrollOptional selector; optional timeoutScroll the matching element into view, or scroll through the page when omitted.
scrapeNoneSave the current page URL and HTML in actions.scrapes.
executeJavascriptscript; optional timeoutEvaluate JavaScript in the page and save its scalar return value.

A request may contain at most 20 actions. Action timeouts default to 10 seconds and cannot exceed 30 seconds. Every action and the final network settle also share the request's overall timeout budget. actions: [] is a no-op: it does not force a browser strategy, and the response contains actions: null.

Combined action results are limited to 25 MiB of serialized UTF-8 JSON. Exceeding this limit aborts the sequence with ok: false and actions: null; results are not truncated, and later actions do not run. This bounds accumulated results, not the browser's memory while producing an individual result.

This example dismisses a cookie banner, waits for the page to update, scrolls to a result, and captures the intermediate page:

{
  "url": "https://example.com/products",
  "formats": ["html", "links", "screenshot"],
  "actions": [
    { "type": "click", "selector": "button[data-accept-cookies]" },
    { "type": "wait", "selector": "main[data-ready]" },
    { "type": "scroll", "selector": "#featured-product" },
    { "type": "scrape" }
  ]
}

The scrape action records the URL and HTML at that exact point. After the complete sequence, Smart Scrape waits for the page to settle and then re-reads the final status, headers, content type, HTML, and URL. Relative links and top-level screenshot/PDF formats therefore describe the final page after navigation or mutation.

The final URL is used internally to resolve relative links; it is not a separate response field. metadata.sourceURL remains the requested URL, while each actions.scrapes[].url identifies the page at that particular scrape action.

executeJavascript runs in the page context with the same network restrictions as BrowserQL. It cannot access Node.js globals or browser-level credential configuration; ordinary page APIs remain available. Scripts are limited to 10,000 characters.

Firecrawl action differences

Smart Scrape does not support a press action; append \n to write.text to send Enter. It also does not support mid-sequence screenshot or pdf actions. Add those values to top-level formats to capture the final page.

{
  "actions": {
    "scrapes": [
      {
        "url": "https://example.com/products",
        "html": "<html>...</html>"
      }
    ],
    "javascriptReturns": [
      { "type": "string", "value": "ready" },
      { "type": "null", "value": null }
    ]
  }
}

Output formats​

The formats array controls what data is returned. The content field contains raw HTML for webpages, parsed JSON for API endpoints, or extracted plain text for PDF targets. Additional formats populate their respective response fields.

Markdown​

Converts the page content to clean markdown, stripping scripts, styles, and non-visible elements.

JSON body:

{
  "url": "https://news.ycombinator.com/",
  "formats": ["markdown"]
}

cURL:

curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["markdown"]}'

Response:

{
  "ok": true,
  "statusCode": 200,
  "content": "<!DOCTYPE html><html>...</html>",
  "markdown": "# Hacker News\n\n[new](newest) | [past](front) | [comments](newcomments)...",
  "screenshot": null,
  "pdf": null,
  "links": null,
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "message": null
}

Raw text​

Returns DOM text with script, style, and noscript elements removed and whitespace collapsed. PDF targets return extracted PDF text. Request it with "formats": ["rawText"]; the result is returned in the rawText response field.

Screenshot​

Returns a full-page screenshot as a base64-encoded PNG string. Including "screenshot" in formats forces a headless browser to be used.

JSON body:

{
  "url": "https://news.ycombinator.com/",
  "formats": ["screenshot"]
}

cURL:

curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["screenshot"]}'

Response:

{
  "ok": true,
  "statusCode": 200,
  "content": "<!DOCTYPE html><html>...</html>",
  "screenshot": "iVBORw0KGgoAAAANSUhEUgAA...",
  "pdf": null,
  "markdown": null,
  "links": null,
  "strategy": "browser",
  "attempted": ["browser"],
  "message": null
}

PDF​

Returns the page as a base64-encoded PDF string. Like "screenshot", including "pdf" forces a headless browser.

JSON body:

{
  "url": "https://news.ycombinator.com/",
  "formats": ["pdf"]
}

cURL:

curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["pdf"]}'

Response:

{
  "ok": true,
  "statusCode": 200,
  "content": "<!DOCTYPE html><html>...</html>",
  "pdf": "JVBERi0xLjQKMSAwIG9iago8PA...",
  "screenshot": null,
  "markdown": null,
  "links": null,
  "strategy": "browser",
  "attempted": ["browser"],
  "message": null
}

Extracts all links (<a href>) from the page, resolves relative URLs to absolute, and filters to http:// and https:// links only.

JSON body:

{
  "url": "https://news.ycombinator.com/",
  "formats": ["links"]
}

cURL:

curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["links"]}'

Response:

{
  "ok": true,
  "statusCode": 200,
  "content": "<!DOCTYPE html><html>...</html>",
  "links": [
    "https://news.ycombinator.com/news",
    "https://news.ycombinator.com/newest",
    "https://news.ycombinator.com/front",
    "https://grapheneos.social/@GrapheneOS/116160393783585567"
  ],
  "screenshot": null,
  "pdf": null,
  "markdown": null,
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "message": null
}

Only scrape main content​

Main content by default

onlyMainContent defaults to true, unless you send includeTags. If you need the full page, set "onlyMainContent": false explicitly.

onlyMainContent removes nav, footer, aside, [role="navigation"], script, style, and noscript elements, so you get the page's main content without menus and boilerplate. It also removes headers, navigation, footers, sidebars, ads, cookie banners, social/share widgets, iframes, scripts, and styles. It affects content, markdown, rawText, and links. The screenshot and pdf formats always capture the full page and aren't affected.

To keep the full page, set it to false:

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://news.ycombinator.com/",
  "formats": ["markdown", "links"],
  "onlyMainContent": false
}'

Explicit onlyMainContent: true can't be combined with includeTags. Sending both returns HTTP 400 with "includeTags" and "onlyMainContent" are mutually exclusive. When you send includeTags without onlyMainContent, it defaults to false.

Content shaping​

For HTML webpages, use includeTags when you want only specific CSS selector matches, or excludeTags when you want the full page minus selected elements. To strip navigation and boilerplate in one step, use onlyMainContent. Each selector list accepts up to 100 entries. Malformed selector entries are ignored and do not return HTTP 400. Valid includeTags matches are still kept when another entry is malformed; Smart Scrape returns unfiltered content only when no include selector matches. Content shaping affects DOM-derived webpage outputs; parsed JSON content and parsed PDF output are unchanged.

onlyMainContent is on by default. Send "onlyMainContent": false to keep the full page; when you send includeTags without onlyMainContent, it defaults to false.

{
  "url": "https://example.com/",
  "formats": ["html", "markdown", "rawText", "links"],
  "onlyMainContent": true,
  "excludeTags": ["nav", ".ad"]
}

Combining includeTags with excludeTags returns HTTP 400: "includeTags" and "excludeTags" are mutually exclusive. For the onlyMainContent conflict, see Only scrape main content.

waitFor is measured in milliseconds after page load and is clamped to the [0, 30000] range. Any positive value forces a browser strategy because an HTTP fetch can't honor a post-load delay.

The headers object sends custom request headers to the target. Browserless first drops host, authorization, proxy-authorization, cookie, set-cookie, x-forwarded-for, x-real-ip, and forwarded. Among the remaining headers, invalid names, non-string values, and values containing CR, LF, or NUL characters return HTTP 400. Custom header names and values can total up to 65,536 bytes.

note

Content shaping affects content, markdown, rawText, and links. It doesn't affect screenshot or pdf; those formats always capture the full rendered page.

PDF targets​

A PDF target is a URL that points directly to an existing PDF. This is different from requesting the "pdf" output format above, which renders a webpage as a new PDF.

Smart Scrape recognizes a PDF when the response uses Content-Type: application/pdf. For responses with a missing content type or application/octet-stream, the final URL path must end in .pdf. The body must also begin with the %PDF- signature. Recognized PDFs handled by the http-fetch and http-proxy strategies have their plain text extracted into content. When "markdown" is requested, markdown contains the extracted text with pages separated by blank lines. Because Smart Scrape does not extract links from inside PDFs, links is null for PDF targets.

Password-protected PDFs, corrupt PDFs, PDFs larger than 25 MiB, PDFs over 10,000 pages, and PDFs whose extracted text exceeds 25 MiB return ok: false with an explanatory message. Requests that force a browser strategy remain unchanged and do not use PDF-target parsing. This includes requests for "screenshot" or "pdf" output and requests using a profile. A PDF target combined with a non-empty actions array returns a controlled failure because browser actions require an HTML page; it never returns Chromium's PDF-viewer HTML as scraped content.

Extract JSON or a summary with AI​

Add "json" or "summary" to formats to have a language model read the scraped page and return structured data or a short prose summary in the same response. Smart Scrape scrapes the page as usual, converts it to markdown, and passes that markdown to the model. Neither format forces a browser, so a page that a plain fetch can serve stays on the cheapest strategy.

Extract structured JSON​

The "json" format needs jsonOptions with a schema, a prompt, or both. The extracted object is returned in the top-level json field.

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE&timeout=60000' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "formats": ["json"],
  "jsonOptions": {
    "schema": {
      "type": "object",
      "properties": {
        "title": { "type": "string" },
        "price": { "type": "number" },
        "currency": { "type": "string" },
        "availability": { "type": "string" },
        "inStockCount": { "type": "integer" },
        "upc": { "type": "string" }
      },
      "required": ["title", "price"]
    },
    "prompt": "Extract the product details. Return price as a number without the currency symbol."
  }
}'

Response (abbreviated):

{
  "ok": true,
  "statusCode": 200,
  "content": "<!DOCTYPE html><html lang=\"en-us\">...</html>",
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "message": null,
  "markdown": null,
  "json": {
    "title": "A Light in the Attic",
    "price": 51.77,
    "currency": "£",
    "availability": "In stock (22 available)",
    "inStockCount": 22,
    "upc": "a897fe39b1053632"
  },
  "summary": null,
  "warning": null
}

markdown is null here even though the model read the page's markdown. Smart Scrape converts the page internally for extraction and only returns markdown when you also list "markdown" in formats.

The schema guides the model but isn't strictly enforced, so validate the object before you rely on it. Fields the page doesn't mention are usually left out rather than invented. Use a prompt without a schema for looser requests such as "List the first 5 books with their prices".

Summarize a page​

The "summary" format returns a few sentences of plain prose in the top-level summary field. It takes no options.

{
  "url": "https://en.wikipedia.org/wiki/Ada_Lovelace",
  "formats": ["summary"]
}

Response (abbreviated):

{
  "ok": true,
  "json": null,
  "summary": "Ada Lovelace (1815–1852) was an English mathematician and writer, the daughter of Lord Byron, best known for her work on Charles Babbage's proposed Analytical Engine. Her notes on the engine include what is often described as the first published computer program.",
  "warning": "Page content was truncated to the first 100000 characters for extraction."
}

You can request "json", "summary", and other formats together. Smart Scrape sends json and summary to the model as two separate calls that run in parallel, and bills both.

What the model sees​

The model only reads the page's markdown. It doesn't see images, styles, or attribute values that markdown drops, so values that only exist visually (for example, a star rating rendered with a CSS class) can't be extracted. Content shaping still applies, so onlyMainContent, includeTags, and excludeTags narrow what the model reads. Text extracted from PDF targets works too.

Page content is treated as untrusted data. The model is instructed to ignore instructions embedded in the page, and only your prompt steers the extraction.

Limits and partial results​

LimitValue
jsonOptions.prompt10,000 characters
jsonOptions.schema20,000 characters when serialized
Page markdown sent to the modelFirst 100,000 characters. Longer pages are truncated and warning says so.
Model output8,000 tokens per format
Time for extractionWhatever remains of the request timeout minus 1.5 seconds, capped at 60 seconds

Extraction shares the request's timeout with the scrape, because the request has a single deadline. If less than 3 seconds remain once the scrape finishes, extraction is skipped. For AI formats, set a larger timeout query parameter, such as timeout=60000.

Extraction problems never fail the scrape. You still get ok: true and every other format you asked for, with json or summary set to null and a warning that explains why:

warningCause
Extraction skipped: not enough of the request timeout remained after scraping. Retry with a larger timeout.Less than 3 seconds of the request timeout remained after scraping.
Extraction skipped: the scraped page had no text content.The page markdown was empty.
Page content was truncated to the first 100000 characters for extraction.The page was longer than the input limit. Extraction still ran on the first 100,000 characters.
Extraction failed: the model did not return a JSON object.The model's output for json wasn't a JSON object.
Extraction failed; the scraped page is returned without it.The json call failed or timed out.
Summary failed; the scraped page is returned without it.The summary call failed or timed out.
Summary failed: the model returned no text.The model returned an empty summary.

When several apply, the messages are joined with spaces. Failed calls aren't retried, so retry the request yourself if you need the result. When the scrape itself fails (ok: false), extraction doesn't run, and json, summary, and warning are all null.

Validation errors​

These requests return HTTP 400 before anything is scraped or billed:

RequestError message
"json" in formats without jsonOptions.schema or a non-blank jsonOptions.prompt"json" format requires "jsonOptions.schema" or "jsonOptions.prompt"
jsonOptions without "json" in formats"jsonOptions" requires "json" in formats
A schema over 20,000 serialized characters"jsonOptions.schema" must not exceed 20000 characters when serialized
An empty formats array"formats" must contain at least one format
"json" or "summary" on a server without an AI providerThe "json" and "summary" formats are not available on this server (no AI provider is configured)

A prompt longer than 10,000 characters, or unknown keys in jsonOptions, fail schema validation with HTTP 400. For pricing, see Unit cost.

Cache results​

Set maxAge to reuse a recent result instead of scraping the same page again. If your account scraped the same URL with the same output options less than maxAge milliseconds ago, Smart Scrape returns that stored result. Otherwise it scrapes the page and stores the result for later requests, unless the request or result is excluded (see What isn't cached). Caching is off by default, because a stale page is the wrong answer for many workloads, so you opt in per request.

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://news.ycombinator.com/",
  "formats": ["markdown"],
  "maxAge": 3600000
}'

The first request scrapes the page and reports a miss (responses abbreviated):

{
  "ok": true,
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "markdown": "# Hacker News\n\n...",
  "metadata": {
    "title": "Hacker News",
    "description": null,
    "language": "en",
    "sourceURL": "https://news.ycombinator.com/",
    "statusCode": 200,
    "cacheState": "miss"
  }
}

The same request within the hour returns the stored result. strategy is "cache", and metadata.cachedAt is when the result was originally scraped:

{
  "ok": true,
  "strategy": "cache",
  "attempted": ["cache"],
  "markdown": "# Hacker News\n\n...",
  "metadata": {
    "title": "Hacker News",
    "description": null,
    "language": "en",
    "sourceURL": "https://news.ycombinator.com/",
    "statusCode": 200,
    "cacheState": "hit",
    "cachedAt": "2026-10-28T09:15:02.481Z"
  }
}

On a hit, strategy is "cache", attempted is ["cache"], message and changeTracking are null, and metadata gains cacheState and cachedAt. The scraped content, including statusCode, headers, content, json, summary, and the rest of metadata, is returned exactly as originally scraped. A hit costs 0.5 units; see Unit cost.

Cache controls​

FieldBehavior
maxAgeMaximum age in milliseconds of a result you'll accept, and how long this request's result is kept. 0 or omitted turns caching off. Values above 48 hours (172800000) are treated as 48 hours. Must be a non-negative integer.
storeInCacheDefaults to true. Set to false to use a cached result if one exists without storing this request's result, for example when you're probing freshness and don't want to overwrite the entry.

Each stored result lives for the maxAge of the request that stored it. A later request with a longer maxAge can't extend it, and a later request with a shorter maxAge ignores entries older than its own limit. JSON clients can also send "3600000" and "false" as strings; they're converted to a number and a boolean.

metadata.cacheState is present only when caching was considered for the request: "hit" when the result came from the cache, and "miss" otherwise. It's absent when maxAge is 0, when the request bypasses the cache, and on failures, where metadata is null.

What makes two requests match​

Results are cached per account, so every API token on the same account shares one cache and other accounts never see your results. Two requests match when they have the same:

  • url
  • formats (order and duplicates don't matter)
  • proxy, onlyMainContent, includeTags, excludeTags, and waitFor
  • jsonOptions, so changing the schema or prompt scrapes and extracts again
  • emulationOs query parameter

The timeout query parameter, strategyCache, maxAge, and storeInCache don't affect matching.

What isn't cached​

These requests bypass the cache entirely. They neither read nor store results, and their responses have no cacheState:

  • requests with a profile query parameter
  • requests with a non-empty actions array
  • requests with custom headers. Headers that Smart Scrape strips anyway, such as authorization or cookie, don't count.
  • requests with "changeTracking" in formats, which always scrape the page

Some results are never stored, so the next matching request scrapes again:

  • failed scrapes (ok: false)
  • pages where the target returned HTTP status 400 or above
  • requests where "json" or "summary" was requested but didn't come back, for example after a timeout or a model error
  • results larger than 1 MiB once compressed, which often includes large screenshots and PDFs

If the cache is temporarily unavailable, Smart Scrape scrapes the page as normal and reports cacheState: "miss". Caching never turns a working request into a failure.

Track changes​

Add "changeTracking" to formats to find out whether a page has changed since you last checked it. Smart Scrape compares the page's markdown with the markdown from your account's last successful check of the same URL and tag, and reports the result in changeTracking. "changeTracking" requires "markdown" in the same formats array. Set changeTrackingOptions.modes to ["git-diff"] to also get the changed lines.

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://example.com/",
  "formats": ["markdown", "changeTracking"],
  "changeTrackingOptions": {
    "modes": ["git-diff"],
    "tag": "daily-check"
  }
}'

The first check of a URL has nothing to compare with, so it reports "new" (responses abbreviated):

{
  "ok": true,
  "strategy": "http-fetch",
  "markdown": "Example Domain\n\nThis domain is for use in documentation examples without needing permission. ...",
  "changeTracking": {
    "changeStatus": "new",
    "previousScrapeAt": null,
    "visibility": "visible",
    "diff": null
  }
}

The next check of an unchanged page reports "same", and previousScrapeAt is when the previous check finished:

{
  "ok": true,
  "strategy": "http-fetch",
  "markdown": "Example Domain\n\nThis domain is for use in documentation examples without needing permission. ...",
  "changeTracking": {
    "changeStatus": "same",
    "previousScrapeAt": "2026-11-02T12:12:51.221Z",
    "visibility": "visible",
    "diff": null
  }
}

When the content differs, the status is "changed". With modes: ["git-diff"], diff holds the changes as a unified diff in text and as structured data in json. This example tracks a page that returns a new ID on every load:

{
  "ok": true,
  "strategy": "http-fetch",
  "markdown": "{ \"uuid\": \"b35f7ce6-fb03-46bc-b280-fc5939df5632\" }",
  "changeTracking": {
    "changeStatus": "changed",
    "previousScrapeAt": "2026-11-02T12:13:04.678Z",
    "visibility": "visible",
    "diff": {
      "text": "===================================================================\n--- previous.md\n+++ current.md\n@@ -1,1 +1,1 @@\n-{ \"uuid\": \"f73a568b-2ccb-439b-b45f-1a80f5732f49\" }\n\\ No newline at end of file\n+{ \"uuid\": \"b35f7ce6-fb03-46bc-b280-fc5939df5632\" }\n\\ No newline at end of file\n",
      "json": {
        "files": [
          {
            "from": "previous.md",
            "to": "current.md",
            "chunks": [
              {
                "content": "@@ -1,1 +1,1 @@",
                "changes": [
                  { "type": "del", "del": true, "ln": 1, "content": "-{ \"uuid\": \"f73a568b-2ccb-439b-b45f-1a80f5732f49\" }" },
                  { "type": "del", "del": true, "ln": 1, "content": "\\ No newline at end of file" },
                  { "type": "add", "add": true, "ln": 1, "content": "+{ \"uuid\": \"b35f7ce6-fb03-46bc-b280-fc5939df5632\" }" },
                  { "type": "add", "add": true, "ln": 1, "content": "\\ No newline at end of file" }
                ]
              }
            ]
          }
        ]
      }
    }
  }
}

Change statuses​

changeStatusMeaning
"new"The page loaded, and there's no history for this URL and tag: it's the first check, or the last successful check was more than 30 days ago. previousScrapeAt is null.
"same"The markdown matches the last successful check. Differences in whitespace and in iframe links are ignored.
"changed"The markdown differs from the last successful check. diff is filled in when you asked for "git-diff".
"removed"The target returned HTTP 404 or 410. The response has ok: false, and previousScrapeAt is the last successful check, or null if there isn't one. This also applies to the very first check of a URL that returns 404 or 410.

previousScrapeAt is when the last successful check finished, whatever its status, not when the page last changed. visibility is always "visible".

changeTracking is null when you didn't request it, and also when Smart Scrape can't give a reliable answer:

  • the scrape failed for a reason other than HTTP 404 or 410, such as a timeout or a page that couldn't be loaded
  • your change history is temporarily unavailable. The scrape is returned as normal, but no status is reported rather than a guessed one.

Diffs​

changeTrackingOptions.modes accepts only "git-diff". With it, diff is set on "changed" responses and holds:

  • text: a unified diff from previous.md (the last successful check) to current.md (this check), with 3 lines of context
  • json.files[].chunks[].changes[]: the same diff as objects. type is "add", "del", or "normal" (an unchanged context line). Added and deleted lines have a line number in ln. Context lines have ln1 (previous) and ln2 (current). content is the line with its diff prefix. A \ No newline at end of file marker can also appear as an add or del entry, so don't count it as a changed line. The status ignores whitespace, but a diff for a "changed" page can still include whitespace-only line edits next to the real changes.

diff is null when the status isn't "changed", when you didn't ask for "git-diff", when either version's markdown is larger than 1 MiB, or when the diff takes longer than 100 ms to compute. The status is still exact in those cases.

How history is kept​

BehaviorValue
ScopeYour account, the URL, and changeTrackingOptions.tag. All API tokens on the account share history, and other accounts never see it. A token that isn't linked to an account keeps its own history, so switching to a different token of that kind starts again with "new".
Retention30 days after the last successful check. Each successful check ("new", "same", or "changed") restarts the 30 days.
Updated bySuccessful checks only. A failed or "removed" check leaves the last successful check in place, so if the page comes back it's compared with the version you saw before it went missing.
Compared contentThe markdown of the page, with whitespace and iframe links ignored
Diff limitsEach version up to 1 MiB of markdown, and up to 100 ms to compute

Use tag to keep separate histories for the same URL, for example one per schedule or per downstream job. A request with a different tag (or no tag) starts its own history with "new". A trailing slash at the very end of the URL doesn't matter (https://example.com/a/ and https://example.com/a share a history). Otherwise the path, query string, and fragment must match exactly, so https://example.com/a/?q=1 and https://example.com/a?q=1 have separate histories.

Only the URL and tag select a history. Other options, such as proxy, onlyMainContent, includeTags, excludeTags, actions, waitFor, and the emulationOs and profile query parameters, don't. Changing how the content is shaped between checks changes the markdown, so it's reported as "changed". Keep those options the same between checks, or use a different tag for each variant.

Change tracking always scrapes the page. With "changeTracking" in formats, maxAge and storeInCache are ignored, nothing is read from or written to the result cache, and metadata has no cacheState.

Change tracking costs nothing extra. The request is billed like the same scrape without it; see Unit cost.

Change tracking validation errors​

These requests return HTTP 400 before anything is scraped or billed:

RequestError message
"changeTracking" in formats without "markdown""changeTracking" requires "markdown" in formats
changeTrackingOptions without "changeTracking" in formats"changeTrackingOptions" requires "changeTracking" in formats
"json" in changeTrackingOptions.modeschangeTracking "json" mode is not supported yet; use "git-diff"
Any other mode except "git-diff""changeTrackingOptions.modes" may only contain "git-diff"
An empty tag"changeTrackingOptions.tag" must be a non-empty string

Unknown keys in changeTrackingOptions, or a modes value that isn't an array, fail schema validation with HTTP 400.

Response fields​

The response examples above are abbreviated. Every response includes all fields listed below.

FieldTypeDescription
okbooleanWhether the scrape succeeded.
statusCodenumber | nullThe HTTP status code from the target site, or null on network errors.
contentstring | object | nullPage content as an HTML string, extracted PDF text, or a parsed JSON object if the target returns application/json. null on failure.
contentTypestring | nullThe content type of the scraped page.
headersobjectHTTP response headers from the target site.
strategystringThe strategy that produced the result (or was being attempted on failure).
attemptedstring[]All strategies attempted, in order.
messagestring | nullError message on failure, null on success.
actionsobject | nullOrdered scrapes and javascriptReturns collected by successful actions. null when actions are omitted, empty, or the scrape fails.
screenshotstring | nullBase64-encoded PNG screenshot, when "screenshot" is in formats.
pdfstring | nullBase64-encoded PDF, when "pdf" is in formats.
markdownstring | nullMarkdown conversion of a webpage, or page-separated extracted text for a PDF target, when "markdown" is in formats.
linksstring[] | nullExtracted links, when "links" is in formats. Always null for PDF targets.
rawTextstring | nullDOM text with scripts/styles removed and whitespace collapsed, or extracted PDF text, when "rawText" is in formats.
metadataobject | nullPage metadata on success, or null on failure. Contains title, description, language, sourceURL (the requested URL, not the final URL after redirects), and statusCode. When caching was considered, it also contains cacheState ("hit" or "miss"), plus cachedAt (ISO 8601 time of the original scrape) on hits.
jsonobject | nullObject extracted per jsonOptions, when "json" is in formats and extraction succeeded. null otherwise.
summarystring | nullAI-written summary, when "summary" is in formats and extraction succeeded. null otherwise.
warningstring | nullWhy requested json or summary output is missing or partial on a successful scrape. null otherwise.
changeTrackingobject | nullWhen "changeTracking" is in formats: changeStatus ("new", "same", "changed", or "removed"), previousScrapeAt (ISO 8601 time of the last successful check, or null), visibility (always "visible"), and diff (text and json, or null). null when not requested, or when no reliable status is available. See Track changes.

JSON auto-parsing​

When the target URL returns JSON content (e.g., an API endpoint with Content-Type: application/json), the content field will contain the parsed JSON object rather than a raw string:

{
  "ok": true,
  "statusCode": 200,
  "content": {
    "userId": 1,
    "id": 1,
    "title": "Example post title",
    "body": "Example post body..."
  },
  "contentType": "application/json; charset=utf-8",
  "strategy": "http-fetch",
  "attempted": ["http-fetch"],
  "message": null
}

Error handling​

On failure, the response still returns HTTP 200 with ok: false and a message describing the error:

{
  "ok": false,
  "statusCode": null,
  "content": null,
  "contentType": null,
  "headers": {},
  "strategy": "browser-captcha",
  "attempted": ["http-fetch", "http-proxy", "browser", "browser-captcha"],
  "message": "Captcha was detected but could not be solved",
  "actions": null,
  "screenshot": null,
  "pdf": null,
  "markdown": null,
  "links": null,
  "rawText": null,
  "metadata": null,
  "json": null,
  "summary": null,
  "warning": null,
  "changeTracking": null
}

Using a profile​

Scrape authenticated pages by passing a saved profile via the ?profile= query parameter. The browser loads the profile's cookies, localStorage, and IndexedDB before navigating, so the page is accessed as the logged-in user.

curl --request POST \
  --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE&profile=acme-prod' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "https://app.example.com/dashboard",
  "formats": ["html", "markdown"]
}'
tip

Create and manage profiles via the Authenticated Profiles workflow. The profile name is scoped to your API token — other tokens cannot access your profiles.

Configuration options​

The /smart-scrape API supports a timeout query parameter to control the maximum time allowed for the scrape operation:

POST /smart-scrape?token=YOUR_API_TOKEN_HERE&timeout=30000

The timeout value is in milliseconds and applies to the entire scrape operation. If not specified, the server default timeout is used.

FAQ & Troubleshooting​

What is the Browserless Smart Scrape API?

Smart Scrape uses a cascading strategy: it tries a fast HTTP fetch first and only launches a full browser if the initial request fails or returns incomplete content. This reduces cost and latency on pages that don't need a full browser.

When should I use Smart Scrape vs the /content API?

Use Smart Scrape when you want automatic fallback from HTTP to browser rendering. Use /content when you know the page requires JavaScript rendering and want to skip the initial HTTP attempt.

Does Smart Scrape handle bot-protected pages?

Yes. When the HTTP fetch is blocked, Smart Scrape falls back to a full browser with stealth capabilities. You can also enable proxies for additional protection.

Why is json null when the scrape succeeded?

Read warning. The most common cause is Extraction skipped: not enough of the request timeout remained after scraping. Extraction shares the request timeout with the scrape, so a slow page can leave too little time for the model. Raise the timeout query parameter, for example to 60000. Other warnings mean the page had no text, the model didn't return a JSON object, or the model call failed. Model calls that error or time out aren't retried and add no extraction units. A call that returns an unusable result (no JSON object, or an empty summary) is still billed for its tokens. Retrying the request scrapes the page again. See Limits and partial results.

Why doesn't my response have cacheState, or why is it always miss?

cacheState is missing when maxAge is 0 or omitted, or when the request uses ?profile=, non-empty actions, custom headers, or the "changeTracking" format, because those requests bypass the cache. A request that always misses usually differs from the earlier one in an option that's part of the match, such as formats or onlyMainContent. It can also mean the earlier result wasn't stored: it failed, the target returned HTTP 400 or above, extraction didn't complete, it exceeded 1 MiB compressed, or it was sent with storeInCache: false.

Why does change tracking report changed when the page looks the same?

Change tracking compares the full markdown, so any text that differs on each load, such as a timestamp, a rotating banner, or a session ID, counts as a change. Narrow the content with onlyMainContent, includeTags, or excludeTags, and keep those options the same on every check, because they aren't part of the history and changing them changes the markdown. Add "modes": ["git-diff"] to see exactly which lines changed. If you get new when you expected a comparison, check that the URL (including its query string) and tag match the earlier check, and that it was less than 30 days ago. See How history is kept.

Next steps​

REST API overview

introduction to all available endpoints

API playground

test endpoints interactively

Was this page helpful?