/scrape
POST/scrape
A JSON-based API that returns text, html, and meta-data from a given list of selectors. Debugging information is available by sending in the appropriate flags in the "debugOpts" property. Responds with an array of JSON objects.
Note: This endpoint is also available at: /chromium/scrape for backwards compatibility.
Request
Query Parameters
Possible values: [datacenter, residential]
Possible values: [0, 1, false, true]
Possible values: [0, 1, false, true]
Body
- Schema
- Example (auto)
- Array [
- ]
- Array [
- ]
- Array [
- CookiePartitionKey
- string
- ]
- Array [
- ]
- PuppeteerLifeCycleEvent[]
- string
- Array [
- ]
- Array [
- Headers
- AnyRecord
- ]
- Array [
- ]
- Array [
- ]
addScriptTag object[]
addStyleTag object[]
authenticate object
Credentials for HTTP authentication. Contains username and password properties.
When bestAttempt is set to true, browserless attempt to proceed when "awaited" events fail or timeout. This includes things like goto, waitForSelector, and more.
cookies object[]
Cookie name.
Cookie value.
The request-URI to associate with the setting of the cookie. This value can affect the default domain, path, and source scheme values of the created cookie.
Cookie domain.
Cookie path.
True if cookie is secure.
True if cookie is http-only.
Cookie SameSite type.
Possible values: [Default, Lax, None, Strict]
Cookie expiration date, session cookie if not set
Cookie Priority. Supported only in Chrome.
Possible values: [High, Low, Medium]
Cookie source scheme type. Supported only in Chrome.
Possible values: [NonSecure, Secure, Unset]
partitionKey object
Cookie partition key. In Chrome, it matches the top-level site the partitioned cookie is available in. In Firefox, it matches the source origin in the {@link https://w3c.github.io/webdriver-bidi/#type-storage-PartitionKey PartitionKey}.
Represents a cookie partition key in Chrome.
The site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.
In Chrome, maps to the CDP's topLevelSite partition key.
Indicates if the cookie has any ancestors that are cross-site to the topLevelSite.
Supported only in Chrome.
debugOpts object
elements object[]required
gotoOptions object
The supported subset of puppeteer's goto options: waitUntil (mapped onto
bap-ts's set — an array reduces to its first entry), timeout (clamped to
the route ceiling), and referer. Other puppeteer goto fields are not
forwarded.
waitUntil object
Possible values: [domcontentloaded, load, networkidle0, networkidle2]
Possible values: [cspviolationreport, document, eventsource, fedcm, fetch, font, image, manifest, media, other, ping, prefetch, preflight, script, signedexchange, stylesheet, texttrack, websocket, xhr]
requestInterceptors object[]
An array of patterns (using req.url().match(r.pattern) to match) and their
corresponding responses to use in order to fulfill those requests.
response objectrequired
headers object
Optional response headers. Either a {@link https://developer.mozilla.org/en-US/docs/Web/API/Headers Headers} instance or a plain record.
The record values will be converted to string following: Arrays' values will be mapped to String (Used when you need multiple headers with the same name). Non-arrays will be converted to String.
A string representation of the body to return. Can be a base64-encoded string but please omit any leading content-type data (eg "data:image/png;base64,").
setExtraHTTPHeaders object
userAgent object
userAgentMetadata object
Used to specify User Agent Client Hints to emulate. See https://wicg.github.io/ua-client-hints Missing optional values will be filled in by the target with what it would normally use.
brands object[]
Brands appearing in Sec-CH-UA.
fullVersionList object[]
Brands appearing in Sec-CH-UA-Full-Version-List.
Used to specify User Agent form-factor values. See https://wicg.github.io/ua-client-hints/#sec-ch-ua-form-factors
viewport object
The viewport dimensions and settings for the page.
Includes properties like width, height, deviceScaleFactor, etc.
The page width in CSS pixels.
The page height in CSS pixels.
Specify device scale factor. See {@link https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio devicePixelRatio} for more info.
Whether the meta viewport tag is taken into account.
Specifies if the viewport is in landscape mode.
Specify if the viewport supports touch events.
waitForEvent object
waitForFunction object
The function, or statement, to be evaluated in browser context
An interval at which the pageFunction is executed, defaults to raf. If polling is a number, then it is treated as an interval in milliseconds at which the function would be executed. If polling is a string, then it can be one of the following values: "raf" or "mutation"
Maximum time to wait for in milliseconds. Defaults to 30000 (30 seconds). Pass 0 to disable timeout.
waitForSelector object
{
"addScriptTag": [
{
"type": "string",
"id": "string",
"url": "string",
"content": "string"
}
],
"addStyleTag": [
{
"url": "string",
"content": "string"
}
],
"authenticate": {
"username": "string",
"password": "string"
},
"bestAttempt": true,
"cookies": [
{
"name": "string",
"value": "string",
"url": "string",
"domain": "string",
"path": "string",
"secure": true,
"httpOnly": true,
"sameSite": "Default",
"expires": 0,
"priority": "High",
"sourceScheme": "NonSecure",
"partitionKey": {
"sourceOrigin": "string",
"hasCrossSiteAncestor": true
}
}
],
"debugOpts": {
"console": true,
"cookies": true,
"html": true,
"network": true,
"screenshot": true
},
"elements": [
{
"selector": "string",
"timeout": 0
}
],
"emulateMediaType": "string",
"gotoOptions": {
"timeout": 0,
"waitUntil": [
"domcontentloaded"
],
"referer": "string"
},
"html": "string",
"rejectRequestPattern": [
"string"
],
"rejectResourceTypes": [
"cspviolationreport"
],
"requestInterceptors": [
{
"pattern": "string",
"response": {
"status": 0,
"headers": {},
"contentType": "string",
"body": "string"
}
}
],
"setExtraHTTPHeaders": {},
"setJavaScriptEnabled": true,
"url": "string",
"userAgent": {
"userAgent": "string",
"userAgentMetadata": {
"brands": [
{
"brand": "string",
"version": "string"
}
],
"fullVersionList": [
{
"brand": "string",
"version": "string"
}
],
"fullVersion": "string",
"platform": "string",
"platformVersion": "string",
"architecture": "string",
"model": "string",
"mobile": true,
"bitness": "string",
"wow64": true,
"formFactors": [
"string"
]
},
"platform": "string"
},
"viewport": {
"width": 0,
"height": 0,
"deviceScaleFactor": 0,
"isMobile": true,
"isLandscape": true,
"hasTouch": true
},
"waitForEvent": {
"event": "string",
"timeout": 0
},
"waitForFunction": {
"fn": "string",
"polling": "string",
"timeout": 0
},
"waitForSelector": {
"hidden": true,
"selector": "string",
"timeout": 0,
"visible": true
},
"waitForTimeout": 0
}
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
Schema
- Fields
- Example (auto)
- Array [
- Array [
- Array [
- ]
- ]
- ]
- Array [
- CookiePartitionKey
- string
- ]
- Array [
- ]
- Array [
- ]
data object[]required
results object[]required
attributes object[]required
debug objectrequired
cookies object[]required
Cookie path.
Cookie expiration date as the number of seconds since the UNIX epoch. Set to -1 for
session cookies
Cookie size.
True if cookie is secure.
True in case of session cookie.
True if cookie partition key is opaque. Supported only in Chrome.
Cookie name.
Cookie value.
Cookie domain.
True if cookie is http-only.
Cookie SameSite type.
Possible values: [Default, Lax, None, Strict]
Cookie Priority. Supported only in Chrome.
Possible values: [High, Low, Medium]
Cookie source scheme type. Supported only in Chrome.
Possible values: [NonSecure, Secure, Unset]
partitionKey object
Cookie partition key. In Chrome, it matches the top-level site the partitioned cookie is available in. In Firefox, it matches the source origin in the {@link https://w3c.github.io/webdriver-bidi/#type-storage-PartitionKey PartitionKey}.
Represents a cookie partition key in Chrome.
The site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.
In Chrome, maps to the CDP's topLevelSite partition key.
Indicates if the cookie has any ancestors that are cross-site to the topLevelSite.
Supported only in Chrome.
network objectrequired
inbound object[]required
outbound object[]required
{
"data": [
{
"results": [
{
"attributes": [
{
"name": "string",
"value": "string"
}
],
"height": 0,
"html": "string",
"left": 0,
"text": "string",
"top": 0,
"width": 0
}
],
"selector": "string"
}
],
"debug": {
"console": [
"string"
],
"cookies": [
{
"path": "string",
"expires": 0,
"size": 0,
"secure": true,
"session": true,
"partitionKeyOpaque": true,
"name": "string",
"value": "string",
"domain": "string",
"httpOnly": true,
"sameSite": "Default",
"priority": "High",
"sourceScheme": "NonSecure",
"partitionKey": {
"sourceOrigin": "string",
"hasCrossSiteAncestor": true
}
}
],
"html": "string",
"network": {
"inbound": [
{
"headers": {},
"status": 0,
"url": "string"
}
],
"outbound": [
{
"headers": {},
"method": "string",
"url": "string"
}
]
},
"screenshot": "string"
}
}
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.