Our first Launch Week is over! See the recap 🚀

Aug 6, 2024

•

Eric Ciarla imageEric Ciarla

Introducing Fire Engine for Firecrawl

Introducing Fire Engine for Firecrawl image

Firecrawl handles web scraping orchestration but doesn’t do the actual scraping. It initially relied on third-party services like Fetch and Playwright for data retrieval. However, these services often failed on certain sites or were too slow, causing issues for users. To address this, we built Fire Engine, now the default backend for Firecrawl. It’s designed to be more reliable and faster, solving the core problems we and our users encountered with other scraping services.

What is Fire Engine?

Fire Engine is a scraping primitive designed to increase Firecrawl’s scraping capabilities.

We’re proud to say that Fire Engine outperforms leading competitors in key areas:

  • Reliability: 40% more reliable than scraping leading competitors when scraping different types of websites

  • Speed: Up to 33.17% faster than scraping leading competitors

And this is just the beginning, we are working closely with Firecrawl users to further improve reliability, speed, and more.

The Technology Behind Fire Engine

Fire Engine combines a variety of browser and non-browser based techniques to balance speed and reliability, ensuring that you get data back without compromise. To do this, Fire engine has:

  • Efficient Headless Browser Management: Running browsers at scale is notoriously difficult, but Fire Engine handles this with ease.
  • Persistent Browser Sessions: By keeping browsers running, Fire Engine improves efficiency when handling new requests, reducing startup times and resource usage.
  • Advanced Web Interaction Techniques: Employing a sophisticated array of methods—including browser-based, browserless, and proprietary approaches
  • Intelligent Request Handling: From smart proxy selection to advanced queuing, every aspect of the request process is optimized for speed and reliability.

With this technology, Fire Engine allows firecrawl to handle millions of requests daily with speed and accuracy.

Try Fire Engine on Firecrawl Today

Fire Engine powers Firecrawl to handle thousands of daily requests efficiently. It’s currently available exclusively through Firecraw cloud, Developers can test Fire Engine’s capabilities by signing up for Firecrawl.

Ready to Build?

Start scraping web data for your AI apps today.
No credit card needed.

About the Author

Eric Ciarla image
Eric Ciarla@ericciarla

Eric Ciarla is the Chief Operating Officer (COO) of Firecrawl and leads marketing. He also worked on Mendable.ai and sold it to companies like Snapchat, Coinbase, and MongoDB. Previously worked at Ford and Fracta as a Data Scientist. Eric also co-founded SideGuide, a tool for learning code within VS Code with 50,000 users.

More articles by Eric Ciarla