/smart-scrape
POST/smart-scrape
Intelligently scrapes a URL using cascading strategies (HTTP fetch, proxy, headless browser, etc).
Request
Query Parameters
Possible values: [phone, tablet]
Possible values: [android, ios, linux, macos, windows]
Possible values: [datacenter, residential]
Possible values: [0, 1, false, true]
Possible values: [0, 1, false, true]
Body
- Schema
- JSON
markdown– page content converted to markdownhtml– cleaned HTML (returned by default incontent, this is a no-op convenience value)rawText– DOM text with script, style, and noscript elements removed and whitespace collapsed; PDF targets return extracted textscreenshot– base64 PNG by default, JPEG whenscreenshotOptions.typeisjpeg(forces browser strategy)pdf– PDF of the page as base64-encoded string (forces browser strategy)links– list of links extracted from the pagejson– AI-extracted object perjsonOptions, without forcing a browsersummary– short AI-written summary, without forcing a browserchangeTracking– compare with the last tracked check; requires markdown and always scrapes freshwaitaccepts exactly one ofmillisecondsorselectorclickaccepts a non-emptyselectorwriteaccepts a non-emptyselectorand stringtextscrolloptionally accepts aselectorscrapecaptures the current page URL and HTMLexecuteJavascriptaccepts a non-emptyscriptup to 10,000 characters
The URL to scrape. Must be an http or https URL.
Output formats to include in the response. Accepts an array of format
strings such as ["markdown", "screenshot"]. When screenshot or pdf
is included, a browser strategy is forced.
Mirrors the Firecrawl "formats" convention.
Possible values: [changeTracking, html, json, links, markdown, pdf, rawText, screenshot, summary]
["html"]changeTrackingOptions object
Options for changeTracking; requires that format and explicit markdown. History is private to the account, URL and tag for 30 days after the last tracked success. Keep content-shaping options consistent between checks; proxy and emulationOs do not select separate histories.
Only git-diff is supported; other modes are rejected by the handler.
Keeps separate history for the same URL.
jsonOptions object
Inputs for the json format; requires json in formats and at least a schema or prompt.
schema object
Best-effort JSON Schema; at most 20,000 characters when serialized.
Best-effort JSON Schema; at most 20,000 characters when serialized.
Instruction guiding the extraction.
Possible values: <= 10000 characters
Browser actions to run in order before the final page is captured. The
supported action types are wait, click, write, scroll, scrape,
and executeJavascript. A maximum of 20 actions is allowed. Detailed,
index-aware validation is performed by the route so the generated schema
intentionally leaves each array entry permissive.
wait, click, write, scroll, and executeJavascript may include a
positive timeout of at most 30,000 milliseconds (default 10,000).
Every action and the final page settle share the request timeout budget.
An empty array is accepted as a no-op. Use a trailing newline in
write.text to submit a form. Screenshots and PDFs are requested through
top-level formats and capture the final page after all actions run.
Possible values: <= 20
The proxy network to route the scrape through. Defaults to residential.
Use datacenter for the cheaper datacenter pool.
Possible values: [datacenter, residential]
location object
Geo-targeting. country is a case-insensitive ISO 3166-1 alpha-2 code.
Forces proxy egress from that country, skipping direct HTTP fetch, and
selects its browser time zone where the existing country mapping supports
it. Omitted preserves today's behavior.
Block ads and trackers in browser tiers.
falsescreenshotOptions object
Screenshot tuning; requires "screenshot" in formats.
Whole page (default) or only the viewport.
trueImage format.
Possible values: [jpeg, png]
pngJPEG quality, integer 1-100. Only valid with type "jpeg".
viewport object
Viewport in CSS pixels, integers 1-4096. Not combinable with emulationOs.
For HTML webpages, extract the main article content, removing navigation,
headers, footers, sidebars and other boilerplate from DOM-derived outputs.
Parsed JSON and PDF content are unchanged. Defaults to true, like Firecrawl
and /crawl, except when includeTags is sent; send false for the full page.
CSS selectors to keep in HTML webpage outputs. Malformed entries are ignored; if no selector matches, unfiltered content is returned. Mutually exclusive with excludeTags and onlyMainContent.
Possible values: <= 100
CSS selectors to remove from HTML webpage outputs. Malformed entries are ignored. Mutually exclusive with includeTags.
Possible values: <= 100
headers object
Custom request headers sent to the target. host, authorization, proxy-authorization, cookie, set-cookie, x-forwarded-for, x-real-ip, and forwarded are removed.
Milliseconds to wait after page load. Clamped to [0, 30000]; positive values force a browser strategy.
0Whether to use and update the remembered winning strategy for this host.
Defaults to true. Set to false to bypass both reads and writes for this
request.
trueReturn a cached result for this account if one younger than this many milliseconds exists for the same url and output options, and cache this request's successful result for up to the same period (at most 48 hours). 0 (default) disables caching. Must be a non-negative integer. Ignored for changeTracking requests and for requests that use profile, non-empty actions, or custom headers that remain after sanitizing (headers that are always stripped do not count).
Possible values: >= 0
0Whether a successful result of a request with maxAge above 0 may be cached for later requests. Set to false to read the cache without adding to it.
true{
"url": "string",
"formats": [
"html",
"links",
"screenshot"
],
"proxy": "datacenter",
"actions": [
{
"type": "click",
"selector": "button[data-accept-cookies]"
},
{
"type": "wait",
"selector": "main[data-ready]",
"timeout": 10000
},
{
"type": "scroll",
"selector": "#featured-product"
},
{
"type": "scrape"
}
]
}
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
An HTML payload of the scraped page content.
Schema
- Fields
- Example (auto)
- AnyRecord
- string
- Array [
- ]
- Array [
- ]
- object
- null
- object
- null
- object
- null
- Array [
- Array [
- Array [
- ]
- ]
- ]
Whether the scrape was successful or not.
The HTTP status code of the response, or null if the scrape failed before an HTTP response was received (e.g. due to a network error or captcha).
content objectrequired
The content of the scraped page. This is extracted text for PDF targets, a string for HTML content, or a parsed object for JSON. Will be null if the scrape was unsuccessful, or if only screenshot/PDF captures were requested and included content exceeded the serialization-work budget.
The content type returned by the target, or null if the scrape was unsuccessful or unavailable. JSON content is parsed into an object; PDF content is extracted into text.
The HTTP response headers returned by the site, or an empty object if unavailable.
The strategy that ultimately succeeded in scraping the page, or the strategy that was being attempted when the scrape failed.
The strategies that were attempted during the scrape, in order.
An error message describing why the scrape failed, or null if the scrape was successful.
A base64 PNG by default, JPEG when screenshotOptions.type is jpeg, present when "screenshot" is in
formats. On a successful scrape this is the scraped page; on a captcha /
anti-bot failure (ok:false) it is a best-effort debug capture of the
blocking page. Null otherwise.
A base64-encoded PDF of the page, present when "pdf" is in formats and the scrape succeeded.
The page content converted to markdown, present when "markdown" is in formats and the scrape succeeded. For PDF targets, this is extracted page text joined with blank lines.
A list of links found on the page, present when "links" is in formats and the scrape succeeded.
actions objectnullablerequired
Values captured by scrape and executeJavascript actions, or null when
actions were omitted, empty, or the scrape failed.
scrapes object[]required
javascriptReturns object[]required
DOM text with scripts/styles removed and whitespace collapsed, or extracted PDF text.
metadata objectrequired
Page metadata on success, or null when scraping fails.
Present only when caching was considered: hit or miss.
Possible values: [hit, miss]
ISO-8601 capture time of the cached result; present only on hits.
json objectrequired
Object extracted per jsonOptions on success; null otherwise.
Object extracted per jsonOptions on success; null otherwise.
AI-written summary when requested and successful; null otherwise.
Why requested extraction is missing or partial on a successful scrape; null otherwise.
changeTracking objectrequired
Null when not requested, history is unavailable, markdown is missing, or the scrape failed other than 404/410. previousScrapeAt is the last successful tracked check, not the last change. A status is never guessed.
Possible values: [changed, new, removed, same]
visiblediff objectrequired
Only for git-diff on changed pages within size/time limits.
json objectrequired
files object[]required
chunks object[]required
changes object[]required
Possible values: [add, del, normal]
{
"ok": true,
"statusCode": 0,
"content": {},
"contentType": "string",
"headers": {},
"strategy": "string",
"attempted": [
"string"
],
"message": "string",
"screenshot": "string",
"pdf": "string",
"markdown": "string",
"links": [
"string"
],
"actions": {
"scrapes": [
{
"url": "string",
"html": "string"
}
],
"javascriptReturns": [
{
"type": "string",
"value": "string"
}
]
},
"rawText": "string",
"metadata": {
"title": "string",
"description": "string",
"language": "string",
"sourceURL": "string",
"statusCode": 0,
"cacheState": "hit",
"cachedAt": "string"
},
"json": {},
"summary": "string",
"warning": "string",
"changeTracking": {
"changeStatus": "changed",
"previousScrapeAt": "string",
"visibility": "visible",
"diff": {
"text": "string",
"json": {
"files": [
{
"from": "string",
"to": "string",
"chunks": [
{
"content": "string",
"changes": [
{
"type": "add",
"content": "string",
"ln": 0,
"ln1": 0,
"ln2": 0,
"add": true,
"del": true,
"normal": true
}
]
}
]
}
]
}
}
}
}
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.