/scrapers
GET/scrapers
Lists the scrapers and APIs available from the loaded scraper bundle, with the bundle version and each API's output schema. Call POST /scrapers/{scraper}/{api} to run one.
Request
Query Parameters
Possible values: [datacenter, residential]
Possible values: [0, 1, false, true]
Possible values: [0, 1, false, true]
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
Schema
- Fields
- Example (auto)
Bundle version (semver only). The build's git sha is never part of a response; operators find it in the logs and the info metric.
Contract version the loaded bundle was built against: one of the versions this host supports, not necessarily its own CONTRACT_VERSION.
Version that was live before the last hot swap (semver only); null until the first swap on this instance.
swap objectrequired
Hot-swap state: whether a replacement bundle is being verified or handshaked right now, which version it is, and how the last swap ended.
scrapers objectrequired
Catalog keyed by scraper name (an organisation slug such as zillow,
never a hostname). Each scraper lists the DNS hosts it handles and its
APIs; each API declares its result-cache TTL and the JSON Schema its
data is validated against on every result.
property name* object
Every DNS host this scraper handles — one organisation may span several.
apis objectrequired
property name* object
Seconds a result for this API stays fresh in the result cache.
Units one call costs. Charged when the call returns HTTP 200 with
ok: true or error code not-found; every other outcome is free.
unitVariants object
Other prices a call may cost, by name, when the call needs more
work (e.g. js: the page rendered in a browser). Absent when the
API has a single price.
outputSchema objectrequired
JSON Schema (draft 2020-12) the API's data is validated against.
The host enforces it on every result.
JSON Schema (draft 2020-12) the API's data is validated against.
The host enforces it on every result.
{
"version": "string",
"contractVersion": 0,
"previousVersion": "string",
"swap": {
"inProgress": true,
"targetVersion": "string",
"startedAt": "string",
"lastOutcome": "string",
"lastAt": "string"
},
"scrapers": {}
}
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.