Introducing Firecrawl Observer, Our Open-Source Website Monitoring Tool

Every business, developer, and researcher has a list of websites they watch. A competitor’s pricing page, a documentation site for a new library, a news outlet for breaking stories. The problem is, checking them is a manual, repetitive chore.

You could use a commercial service, but they often come with high costs, strict limits, and little flexibility. They are black boxes where you put in a URL and hope for the best.

So we asked ourselves: What if you could build a powerful, transparent, and open-source tool to monitor any website for any change, exactly how you want?

This is the story of Firecrawl Observer.

Firecrawl Observer is a complete, self-hostable website monitoring platform. It’s the latest addition to our open-source AI toolbox, designed to give you a reliable eye on the web:

1. Enter any URL you want to monitor.
2. Set your schedule, from every few seconds to once a week.
3. Get notified via email or webhook the moment a change is detected.
4. See exactly what’s different with a visual, side-by-side diff viewer.

It’s a complete system for turning the dynamic web into a structured stream of events, ready to be integrated into any workflow.

Firecrawl Observer is built on a modern, serverless stack designed for reliability and scale.

• Smart Scraping with Firecrawl: At its core, Observer uses Firecrawl to reliably scrape and structure data from any website. It leverages Firecrawl’s built-in changeTracking to get clean markdown and an intelligent diff of what’s new, what’s gone, and what’s the same. This bypasses the need for complex, site-specific parsers.

• Real-time Backend with Convex: All data—websites, scrape results, user accounts, and change alerts—is stored in Convex. Its serverless functions and schedulers handle the monitoring logic, running checks at your specified intervals and storing the results automatically.

• AI-Powered Analysis: This is where it gets smart. Instead of alerting you for every trivial change (like a new cookie banner or a rotating ad), Observer can use AI to determine if a change is actually meaningful. By connecting your OpenAI key, it sends the diff to a model (like GPT-4o-mini) which scores the change and provides a reason, letting you filter out the noise.

• Instant Notifications: When a significant change is detected, Observer springs into action. It can send a detailed email with a summary of the changes or fire a webhook with a structured JSON payload, ready for integration with Slack, Discord, or your own internal systems.

Website monitoring is a deep field. Our goal with Observer isn’t to be a closed, one-size-fits-all solution.

Instead, we want to build a powerful, open-source foundation that anyone can use, understand, and build upon.

The version you see today is a complete application, but we know there’s always more to build. By open-sourcing it, we’re inviting you to help shape its future.

• Want to add a new notification channel? Fork the repo and add a new provider.
• Think the AI prompt can be better? Open a pull request.
• Have an idea for a new feature, like CSS selector monitoring? Start a discussion in the issues.

We believe that by building in public, we can create a tool that is not only powerful and free but also more robust and adaptable, driven by the collective intelligence of the open-source community.

We invite you to explore Firecrawl Observer. Use it to track your competitors, monitor product availability, or just keep an eye on your favorite sites.

• Explore the code: View on GitHub

We built Firecrawl Observer to democratize website monitoring. No more expensive subscriptions or black-box services. Just clone, deploy, and start observing.

Let’s build a more transparent web. Together.