PAI Chat·Website crawling

Website crawling

Instead of uploading files one by one, point a chatbot at a URL and let it crawl your site into the knowledge base. The crawler walks same-host pages breadth-first, extracting page text and PDFs for embedding.

Starting a crawl

POST a start url to the chatbot. The crawler does a breadth-first walk of pages on the same host. Narrow the scope with includePatterns / excludePatterns (regex over URL paths). Exclude patterns always win over include patterns. This is an admin operation — use an organization key.

cURL

curl https://chat-api-dev.paicloud.ai//chatbot/CHATBOT_ID/web/crawl \
  -H "x-api-key: $PAI_CHAT_API_KEY" \
  -H "x-organization-id: YOUR_ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "excludePatterns": ["/blog/.*", ".*\\?print=1"],
    "includePatterns": ["/docs/.*"]
  }'

The crawl runs asynchronously. The response returns the crawl request with a status and, once finished, pagesFound and pdfsFound counts:

200 response

{
  "request": {
    "id": "…",
    "chatbotId": "…",
    "startUrl": "https://example.com/docs",
    "status": "pending",
    "pagesFound": null,
    "pdfsFound": null,
    "createdAt": "…"
  }
}

Scoping with patterns

includePatterns — if set, a URL must match at least one to be crawled.
excludePatterns — matched paths are skipped; takes precedence over includes. Sensible defaults apply if omitted.

Managing crawls

Alongside starting a crawl you can:

List a chatbot's crawl requests and their status.
Recrawl to refresh content from a previous run without re-entering its configuration.
Sitemap — seed the crawl from a site's sitemap for fuller coverage.
Delete a crawl request and its ingested pages.

schedule

Crawls are asynchronous — poll the crawl request until status settles. Crawled pages become retrievable once embedded, the same as uploaded documents.

gpp_maybe

Only crawl sites you own or are authorised to index. The crawler stays on the start host; use exclude patterns to avoid login walls, search pages, and infinite parameter loops.