Introducing Smart Crawl! Join the waitlist to turn any website into an API with AI

June 17, 2024

Nicolas Camara imageNicolas Camara

Firecrawl x Dify.AI: Add web data to your RAG applications

Firecrawl x Dify.AI: Add web data to your RAG applications image

We are excited to announce that Firecrawl now integrates seamlessly with Dify, empowering your RAG (Retrieval-Augmented Generation) applications like never before!

Introducing Dify x Firecrawl

Dify is an open-source platform for developing LLM (Large Language Model) applications. It allows you to orchestrate everything from simple agents to complex AI workflows, all powered by a robust RAG engine. With Firecrawl, you can now add web data to your RAG applications - all inside Dify’s platform.

Ingesting Web Data

Firecrawl enables you to convert any website into clean, LLM-ready data effortlessly. To use it in Dify, configure your settings in the Knowledge dashboard. There you can set your Firecrawl API key and start crawing right away.

Easy Setup and Customization

To start using it, pass the URL you want to ingest by clicking the Sync from website button in the Knowledge section. Tailor your crawling process with options to set sub-page limits and depths. Use the Exclude and Include path options to ensure you capture exactly the data you need.

Data Embedding

Firecrawl rapidly crawls web pages in parallel. After crawling, you can select the desired web data on Dify for preprocessing and cleaning. The processed data is then embedded and stored in Dify’s vector DB as a new knowledge base.

Ready for production

Now, you can create a RAG app that uses web data as contextual knowledge on Dify with the power of Firecrawl. We value your feedback! Connect with us on X/Twitter @firecrawl_dev.

Ready to Build?

Start scraping web data for your AI apps today.
No credit card needed.