v1.4.4: PDF and image sub-link extraction

v1.4.4 Release

Features & Enhancements

  • Scrape API: Added action & wait time validation (#1146)
  • Extraction Improvements:
    • Added detection of PDF/image sub-links & extracted text via Gemini (#1173)
    • Multi-entity prompt enhancements for extraction (#1181)
    • Show sources out of __experimental in extraction (#1180)
  • Environment Setup: Added Serper & Search API env vars to docker-compose (#1147)
  • Credit System Update: Now displays "tokens" instead of "credits" when out of tokens (#1178)

Examples

Fixes

  • HTML Transformer: Updated free_string function parameter type (#1163)
  • Gemini Crawler: Updated library & improved PDF link extraction (#1175)
  • Crawl Queue Worker: Only reports successful page count in num_docs (#1179)
  • Scraping & URLs:
    • Fixed relative URL conversion (#584)
    • Enforced scrape rate limit in batch scraping (#1182)