Website crawler & knowledge sync
Queue safe same-host crawls that discover pages, extract readable text, create chunks and build embeddings for the selected website.
Create a crawl job
Open Crawler, enter a public HTTP/HTTPS URL, select Page, Website or Sitemap mode, then choose a page limit. Jobs run through the crawler worker so the Admin request stays fast.
Run the worker
php bin/process-crawls.php 25
Use Cron or a process supervisor in production. Review job status, discovered pages and indexed pages from the Crawler page.
Isolation & safety
The crawler rejects localhost and private/reserved network targets, limits response size and indexes content only into the selected website knowledge base.