For AI agents: a documentation index is available at /llms.txt
Skip to main content

/crawl/*

GET 

/crawl/*

Get the status of a crawl job and its paginated page results. Returns progress counters, per-page metadata with content URLs, and a next URL for offset-based pagination via the skip parameter.

Request​

Query Parameters

    skip number
    token string

Responses​

Schema
    statusCrawlStatusrequired

    Possible values: [cancelled, completed, failed, in-progress]

    totalnumberrequired
    completednumberrequired
    failednumberrequired
    expiresAtstringrequired
    nextstringrequired
    data object[]required
  • Array [
  • statusPageStatusrequired

    Possible values: [cancelled, completed, failed, in-progress, queued]

    contentUrlstringrequired
    metadata objectrequired
    titlestringrequired
    descriptionstringrequired
    languagestringrequired
    scrapedAtstringrequired
    sourceURLstringrequired
    statusCodenumberrequired
    errorstringrequired
  • ]
Was this page helpful?