DOCS
Dashboardnorth_eastGet an API keyvpn_key
PAI Chat·Endpoint
POST/chatbot/{chatbotId}/web/crawl

Crawl a website starting from a URL. BFS through same-host pages, excluding matc

Crawl a website starting from a URL. BFS through same-host pages, excluding matched patterns.

DELETE/chatbot/{chatbotId}/web/crawl/{requestId}arrow_forward
tune
Parameters
chatbotIdrequiredpath
string (uuid)
The ID of the chatbot
upload
Request body
urlrequired
string (uri)
Starting URL to crawl from
excludePatterns
array
Regex patterns for URL paths to exclude. Defaults applied if omitted.
includePatterns
array
Regex patterns for URL paths to include. If set, a URL must match at least one to be crawled. Exclude patterns still take precedence.
double_arrow
Returns 200
requestrequired
object
idrequired
string (uuid)
chatbotIdrequired
string (uuid)
startUrlrequired
string
excludePatterns
nullarray
includePatterns
nullarray
statusrequired
string
pagesFound
nullnumber
pdfsFound
nullnumber
error
nullstring
createdAtrequired
string (date-time)
completedAt
null,string (date-time)
lastCrawledAt
null,string (date-time)
report
Errors
StatusDescription
400Default Response
500Default Response
north_east
Request
curl -X POST 'https://chat-api-dev.paicloud.ai//chatbot/{chatbotId}/web/crawl' \
  -H 'Authorization: Bearer YOUR_API_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "url",
  "excludePatterns": [],
  "includePatterns": []
}'
south_west
Response
200 OK
application/json
{
  "request": {
    "id": "00000000-0000-0000-0000-000000000001",
    "chatbotId": "00000000-0000-0000-0000-000000000001",
    "startUrl": "startUrl",
    "excludePatterns": "excludePatterns",
    "includePatterns": "includePatterns",
    "status": "status",
    "pagesFound": "pagesFound",
    "pdfsFound": "pdfsFound",
    "error": "error",
    "createdAt": "2026-01-01T00: 00:00Z",
    "completedAt": "2026-01-01T00: 00:00Z",
    "lastCrawledAt": "2026-01-01T00: 00:00Z"
  }
}