Crawl reliability fixes

Crawl reliability improvements

  • Fixed 500 errors that would happen often in some crawled websites and when servers were at capacity
  • Fixed an issue where v1 crawl status wouldn't properly return pages over 10mb
  • Fixed an issue where screenshot would return undefined
  • Push improvements that reduce speed times when a scraper fails