/crawl/*
GET/crawl/*
Get the status of a crawl job and its paginated page results. Returns progress counters, per-page metadata with content URLs, and a next URL for offset-based pagination via the skip parameter.
Request
Query Parameters
skip number
token string
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
Schema
- Fields
- Example (auto)
- Array [
- ]
statusCrawlStatusrequired
Possible values: [cancelled, completed, failed, in-progress]
totalnumberrequired
completednumberrequired
failednumberrequired
expiresAtstringrequired
nextstringrequired
data object[]required
statusPageStatusrequired
Possible values: [cancelled, completed, failed, in-progress, queued]
contentUrlstringrequired
metadata objectrequired
titlestringrequired
descriptionstringrequired
languagestringrequired
scrapedAtstringrequired
sourceURLstringrequired
statusCodenumberrequired
errorstringrequired
{
"status": "cancelled",
"total": 0,
"completed": 0,
"failed": 0,
"expiresAt": "string",
"next": "string",
"data": [
{
"status": "cancelled",
"contentUrl": "string",
"metadata": {
"title": "string",
"description": "string",
"language": "string",
"scrapedAt": "string",
"sourceURL": "string",
"statusCode": 0,
"error": "string"
}
}
]
}
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.
Was this page helpful?