Self-host overhaul: v1.5.0

v1.5.0 Release

Self-Host Fixes

  • Reworked Guide: The SELF_HOST.md and docker-compose.yaml have been updated for clarity and compatibility
  • Kubernetes Improvements: Updated self-hosted Kubernetes deployment examples for compatibility and consistency (#1177)
  • Self-Host Fixes: Numerous fixes aimed at improving self-host performance and stability (#1207)
  • Proxy Support: Added proxy support tailored for self-hosted environments (#1212)
  • Playwright Integration: Added fixes and continuous integration for the Playwright microservice (#1210)
  • Search Endpoint Upgrade: Added SearXNG support for the /search endpoint (#1193)

Core Fixes & Enhancements

  • Crawl Status Fixes: Fixed various race conditions in the crawl status endpoint (#1184)
  • Timeout Enforcement: Added timeout for scrapeURL engines to prevent hanging requests (#1183)
  • Query Parameter Retention: Map function now preserves query parameters in results (#1191)
  • Screenshot Action Order: Ensured screenshots execute after specified actions (#1192)
  • PDF Scraping: Improved handling for PDFs on complex websites (#1198)
  • Map/scrapeURL Abort Control: Integrated AbortController to stop scraping when the request times out (#1205)
  • SDK Timeout Enforcement: Enforced request timeouts in the SDK (#1204)

New Features & Additions

  • Proxy Options: Introduced proxy configuration options for infrastructure management (#1196)
  • Deep Research (Alpha): Launched an alpha implementation of deep research (#1202)
  • LLM Text Generator: Added a new endpoint for llms.txt generation (#1201)

Docker & Containerization

  • Production Ready Docker Image: A streamlined, production ready Docker image is now available to simplify self-hosted deployments.