For AI agents: a documentation index is available at /llms.txt
Skip to main content

/scrape

POST 

/scrape

A JSON-based API that returns text, html, and meta-data from a given list of selectors. Debugging information is available by sending in the appropriate flags in the "debugOpts" property. Responds with an array of JSON objects.

Note: This endpoint is also available at: /chromium/scrape for backwards compatibility.

Request​

Query Parameters

    allowedDomains string[]
    blockAds boolean
    blockAdsInclude string
    credentials object
    externalProxyServer string
    integrationId string
    launch object
    profile string
    proxy ProxyType

    Possible values: [datacenter, residential]

    proxyCity string
    proxyCountry string
    proxyLocaleMatch string

    Possible values: [0, 1, false, true]

    proxyPreset string
    proxyState string
    proxySticky string

    Possible values: [0, 1, false, true]

    timeout number
    token string
    trackingId string

Body

    addScriptTag object[]
  • Array [
  • typestring
    idstring
    urlstring
    contentstring
    property name*any
  • ]
  • addStyleTag object[]
  • Array [
  • urlstring
    contentstring
    property name*any
  • ]
  • authenticate object

    Credentials for HTTP authentication. Contains username and password properties.

    usernamestringrequired
    passwordstringrequired
    bestAttemptboolean

    When bestAttempt is set to true, browserless attempt to proceed when "awaited" events fail or timeout. This includes things like goto, waitForSelector, and more.

    cookies object[]
  • Array [
  • namestringrequired

    Cookie name.

    valuestringrequired

    Cookie value.

    urlstring

    The request-URI to associate with the setting of the cookie. This value can affect the default domain, path, and source scheme values of the created cookie.

    domainstring

    Cookie domain.

    pathstring

    Cookie path.

    secureboolean

    True if cookie is secure.

    httpOnlyboolean

    True if cookie is http-only.

    sameSitestring

    Cookie SameSite type.

    Possible values: [Default, Lax, None, Strict]

    expiresnumber

    Cookie expiration date, session cookie if not set

    prioritystring

    Cookie Priority. Supported only in Chrome.

    Possible values: [High, Low, Medium]

    sourceSchemestring

    Cookie source scheme type. Supported only in Chrome.

    Possible values: [NonSecure, Secure, Unset]

    partitionKey object

    Cookie partition key. In Chrome, it matches the top-level site the partitioned cookie is available in. In Firefox, it matches the source origin in the {@link https://w3c.github.io/webdriver-bidi/#type-storage-PartitionKey PartitionKey}.

    anyOf

    Represents a cookie partition key in Chrome.

    sourceOriginstringrequired

    The site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.

    In Chrome, maps to the CDP's topLevelSite partition key.

    hasCrossSiteAncestorboolean

    Indicates if the cookie has any ancestors that are cross-site to the topLevelSite.

    Supported only in Chrome.

  • ]
  • debugOpts object
    consoleboolean
    cookiesboolean
    htmlboolean
    networkboolean
    screenshotboolean
    elements object[]required
  • Array [
  • selectorstringrequired
    timeoutnumber
  • ]
  • emulateMediaTypestring
    gotoOptions object

    The supported subset of puppeteer's goto options: waitUntil (mapped onto bap-ts's set — an array reduces to its first entry), timeout (clamped to the route ceiling), and referer. Other puppeteer goto fields are not forwarded.

    timeoutnumber
    waitUntil object
    anyOf
  • Array [
  • PuppeteerLifeCycleEvent

    Possible values: [domcontentloaded, load, networkidle0, networkidle2]

  • ]
  • refererstring
    htmlstring
    rejectRequestPatternstring[]
    rejectResourceTypesstring[]

    Possible values: [cspviolationreport, document, eventsource, fedcm, fetch, font, image, manifest, media, other, ping, prefetch, preflight, script, signedexchange, stylesheet, texttrack, websocket, xhr]

    requestInterceptors object[]
  • Array [
  • patternstringrequired

    An array of patterns (using req.url().match(r.pattern) to match) and their corresponding responses to use in order to fulfill those requests.

    response objectrequired
    statusnumber
    headers object

    Optional response headers. Either a {@link https://developer.mozilla.org/en-US/docs/Web/API/Headers Headers} instance or a plain record.

    The record values will be converted to string following: Arrays' values will be mapped to String (Used when you need multiple headers with the same name). Non-arrays will be converted to String.

    anyOf
    object
    contentTypestring
    bodystring

    A string representation of the body to return. Can be a base64-encoded string but please omit any leading content-type data (eg "data:image/png;base64,").

  • ]
  • setExtraHTTPHeaders object
    property name*string
    setJavaScriptEnabledboolean
    urlstring
    userAgent object
    userAgentstring
    userAgentMetadata object

    Used to specify User Agent Client Hints to emulate. See https://wicg.github.io/ua-client-hints Missing optional values will be filled in by the target with what it would normally use.

    brands object[]

    Brands appearing in Sec-CH-UA.

  • Array [
  • brandstringrequired
    versionstringrequired
  • ]
  • fullVersionList object[]

    Brands appearing in Sec-CH-UA-Full-Version-List.

  • Array [
  • brandstringrequired
    versionstringrequired
  • ]
  • fullVersionstring
    platformstringrequired
    platformVersionstringrequired
    architecturestringrequired
    modelstringrequired
    mobilebooleanrequired
    bitnessstring
    wow64boolean
    formFactorsstring[]

    Used to specify User Agent form-factor values. See https://wicg.github.io/ua-client-hints/#sec-ch-ua-form-factors

    platformstring
    viewport object

    The viewport dimensions and settings for the page. Includes properties like width, height, deviceScaleFactor, etc.

    widthnumberrequired

    The page width in CSS pixels.

    heightnumberrequired

    The page height in CSS pixels.

    deviceScaleFactornumber

    Specify device scale factor. See {@link https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio devicePixelRatio} for more info.

    isMobileboolean

    Whether the meta viewport tag is taken into account.

    isLandscapeboolean

    Specifies if the viewport is in landscape mode.

    hasTouchboolean

    Specify if the viewport supports touch events.

    waitForEvent object
    eventstringrequired
    timeoutnumber
    waitForFunction object
    fnstringrequired

    The function, or statement, to be evaluated in browser context

    pollingstring | number

    An interval at which the pageFunction is executed, defaults to raf. If polling is a number, then it is treated as an interval in milliseconds at which the function would be executed. If polling is a string, then it can be one of the following values: "raf" or "mutation"

    timeoutnumber

    Maximum time to wait for in milliseconds. Defaults to 30000 (30 seconds). Pass 0 to disable timeout.

    waitForSelector object
    hiddenboolean
    selectorstringrequired
    timeoutnumber
    visibleboolean
    waitForTimeoutnumber

Responses​

Schema
    data object[]required
  • Array [
  • results object[]required
  • Array [
  • attributes object[]required
  • Array [
  • namestringrequired
    valuestringrequired
  • ]
  • heightnumberrequired
    htmlstringrequired
    leftnumberrequired
    textstringrequired
    topnumberrequired
    widthnumberrequired
  • ]
  • selectorstringrequired
  • ]
  • debug objectrequired
    consolestring[]required
    cookies object[]required
  • Array [
  • pathstringrequired

    Cookie path.

    expiresnumberrequired

    Cookie expiration date as the number of seconds since the UNIX epoch. Set to -1 for session cookies

    sizenumberrequired

    Cookie size.

    securebooleanrequired

    True if cookie is secure.

    sessionbooleanrequired

    True in case of session cookie.

    partitionKeyOpaqueboolean

    True if cookie partition key is opaque. Supported only in Chrome.

    namestringrequired

    Cookie name.

    valuestringrequired

    Cookie value.

    domainstringrequired

    Cookie domain.

    httpOnlyboolean

    True if cookie is http-only.

    sameSitestring

    Cookie SameSite type.

    Possible values: [Default, Lax, None, Strict]

    prioritystring

    Cookie Priority. Supported only in Chrome.

    Possible values: [High, Low, Medium]

    sourceSchemestring

    Cookie source scheme type. Supported only in Chrome.

    Possible values: [NonSecure, Secure, Unset]

    partitionKey object

    Cookie partition key. In Chrome, it matches the top-level site the partitioned cookie is available in. In Firefox, it matches the source origin in the {@link https://w3c.github.io/webdriver-bidi/#type-storage-PartitionKey PartitionKey}.

    anyOf

    Represents a cookie partition key in Chrome.

    sourceOriginstringrequired

    The site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.

    In Chrome, maps to the CDP's topLevelSite partition key.

    hasCrossSiteAncestorboolean

    Indicates if the cookie has any ancestors that are cross-site to the topLevelSite.

    Supported only in Chrome.

  • ]
  • htmlstringrequired
    network objectrequired
    inbound object[]required
  • Array [
  • headersobjectrequired
    statusnumberrequired
    urlstringrequired
  • ]
  • outbound object[]required
  • Array [
  • headersobjectrequired
    methodstringrequired
    urlstringrequired
  • ]
  • screenshotstringrequired
Was this page helpful?