Z
member badge
Zy
Business Problem:
Currently, users have limited control over how website content is crawled for the Knowledge Source feature. This can result in unwanted pages being indexed, inefficient crawling of deeply nested content, and unnecessary reliance on sitemaps that may not always be optimal. Without configuration options, users face challenges with crawl efficiency, content relevance, and overall crawl performance.
Desired Outcome:
Provide advanced crawl configuration options to give users greater control over the crawling process. Specifically:
  • Set Maximum Crawl Depth: Allow users to specify how deep the crawler should follow links from the seed URL.
  • Define URL Exclusion Rules: Enable users to create rules or patterns (e.g., path, query string) to exclude specific URLs from being crawled and indexed.
  • Enable or Disable Sitemap-Based Crawling: Give users the choice to enable or disable sitemap-based crawling.
These enhancements will improve crawl efficiency, content quality, and user confidence in the Knowledge Source feature.