Launch Week I / Day 7: Crawl Webhooks (v1)
Welcome to Day 7 of Firecrawl’s Launch Week! We’re excited to introduce new /crawl webhook support.
Crawl Webhook
You can now pass a webhook
parameter to the /crawl
endpoint. This will send a POST request to the URL you specify when the crawl is started, updated and completed.
The webhook will now trigger for every page crawled and not just the whole result at the end.
Webhook Events
There are now 4 types of events:
crawl.started
- Triggered when the crawl is started.crawl.page
- Triggered for every page crawled.crawl.completed
- Triggered when the crawl is completed to let you know it’s done.crawl.failed
- Triggered when the crawl fails.
Webhook Response
success
- If the webhook was successful in crawling the page correctly.type
- The type of event that occurred.id
- The ID of the crawl.data
- The data that was scraped (Array). This will only be non empty oncrawl.page
and will contain 1 item if the page was scraped successfully. The response is the same as the/scrape
endpoint.error
- If the webhook failed, this will contain the error message.
Learn More
Learn more about the webhook in our documentation.
Ready to Build?
Start scraping web data for your AI apps today.
No credit card needed.
About the Author
Nicolas Camara is the Chief Technology Officer (CTO) at Firecrawl. He previously built and scaled Mendable, one of the pioneering "chat with your documents" apps, which had major Fortune 500 customers like Snapchat, Coinbase, and MongoDB. Prior to that, Nicolas built SideGuide, the first code-learning tool inside VS Code, and grew a community of 50,000 users. Nicolas studied Computer Science and has over 10 years of experience in building software.
More articles by Nicolas Camara
Extract website data using LLMs
Learn how to use Firecrawl and Groq to extract structured data from a web page in a few lines of code.
Firecrawl x Dify.AI: Add web data to your RAG applications
Learn how Firecrawl integrates with Dify.AI to provide web data to AI applications.
Launch Week I / Day 6: LLM Extract (v1)
Extract structured data from your web pages using the extract format in /scrape.
Launch Week I / Day 7: Crawl Webhooks (v1)
New /crawl webhook support. Send notifications to your apps during a crawl.
Build a 'Chat with website' using Groq Llama 3
Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.
Scrape and Analyze Airbnb Data with Firecrawl and E2B
Learn how to scrape and analyze Airbnb data using Firecrawl and E2B in a few lines of code.
Firecrawl June 2024 Updates
Discover the latest features, integrations, and improvements in Firecrawl for June 2024.