How to Monitor a Website (or the Entire Web) for Changes with Firecrawl

Keeping up with a fast-moving page, like a changelog, a pricing page, or the news cycle around a specific AI model, usually means refreshing a tab every few minutes. Firecrawl's monitor endpoint replaces that: it scrapes a page (or runs a search) on a schedule you set, diffs the result against the last snapshot, and only pings you when something actually changed.

Let's go through how to use it.

Unlike scrape, search, interact, and parse, which all work keyless on a rate-limited free tier (see the rate limits docs), the monitor endpoint always requires an API key. Sign up at firecrawl.dev to get one, along with free credits and a higher rate limit than the keyless tier offers.

This walkthrough uses the Node SDK, but monitors can also be created from the Python SDK, the CLI, the MCP server, or directly in the Firecrawl dashboard.

Install the SDK with npm:

npm install firecrawl

Then set up the client and create a monitor. The example below watches the GitHub releases page for CamKit, an open-source project, so it fires whenever a new version is published:

import { Firecrawl } from "firecrawl";
 
const fc = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
 
const monitor = await fc.createMonitor({
  name: "Camkit releases",
  schedule: { text: "every 5 minutes", timezone: "UTC" },
  targets: [
    {
      type: "scrape",
      urls: ["https://github.com/Orva-Studio/camkit/releases"],
      scrapeOptions: {},
    },
  ],
  goal: "Notify me when a new camkit release is published",
  webhook: { url: process.env.WEBHOOK_URL as string },
});
 
console.log(monitor.id);

But I'll break it down so it's easy to understand.

There are three required inputs:

• name is a label for the monitor so you can identify it later.
• schedule accepts either cron syntax or natural-language text like "every 5 minutes", "hourly", or "daily at 9am", the minimum interval is 5 minutes.
• targets is what gets watched; a monitor accepts 1 to 50 targets and can mix types, but it's best to keep one monitor per concern rather than bundling unrelated URLs together.

Here the target has type: "scrape" and a urls array, meaning it watches known pages. scrapeOptions passes straight through to the underlying scrape, so the usual formats (markdown, html, summary, or a json format with its own schema) all apply, scrapeOptions: {} here just means the defaults are used.

Two fields are optional but do most of the work:

• goal tells Firecrawl's AI judge exactly what counts as a meaningful change, so it can ignore everything else.
• webhook is where Firecrawl posts a notification whenever a meaningful change is detected. Email notifications work the same way and can be configured instead of or alongside a webhook.

Running the file above returns the new monitor's ID. If the Firecrawl CLI is installed, you can list monitors and pipe the output through jq (if you have it installed) for a readable view:

firecrawl monitor list | jq .

The output includes the normalized cron ("every 5 minutes" became */5 * * * *), status, and nextRunAt, everything needed to confirm the monitor is set up the way you expect without opening the dashboard. If you've signed up for Firecrawl, the same monitor also shows up in the Firecrawl dashboard, with its schedule, estimated monthly credit cost, and recent checks.

To see the monitor in action, point its webhook at a small local server (the demo here uses a Bun server running on port 3000, tunneled out with a Tailscale funnel so it doesn't need a public deploy) and send it a test request:

curl -s -X POST https://richards-macbook-pro.tail45f396.ts.net/ \
  -H 'content-type: application/json' \
  -d '{"test":"hello from funnel"}'

The first real check on the target repo comes back showing no meaningful differences, since nothing has changed yet, and the Firecrawl dashboard reflects the same result.

When a new version is published to the watched repo, the monitor has something real to catch. The next check marks the page as changed, flags it as meaningful, and includes the actual diff, the old version string next to the new one:

This is the AI judge doing its job: it only marked this check as meaningful because the change matched the goal ("a new release is published"). A run that doesn't match the goal still happens and still costs a scrape credit, but it won't trigger a notification or add a judge credit.

A monitor's output isn't just a notification, it's structured data your own code can act on. In the demo, a fake course site for a CLI tool called CamKit displays the tool's latest version number, and manually updating that number every release is exactly the kind of busywork a monitor removes:

The webhook server keeps that version in sync with plain string matching, no extra Firecrawl calls needed. For each monitor.page payload, it skips pages where isMeaningful is false, then scans the added lines in diff.text for a GitHub release-tag pattern (releases/tag/v1.2.3), collects every version it finds, and keeps the highest one by semver:

function extractNewVersion(payload) {
  if (payload?.type !== "monitor.page") return null;
 
  const versions = [];
  // Each Firecrawl monitor check carries an isMeaningful flag and a diff to read
  for (const check of payload.data ?? []) {
    if (!check.isMeaningful) continue;
 
    // Look at added lines in the diff for a new release tag
    for (const line of (check.diff?.text ?? "").split("\n")) {
      if (!line.startsWith("+")) continue;
      const match = line.match(/releases\/tag\/(v\d+\.\d+\.\d+)/);
      if (match) versions.push(match[1]);
    }
  }
  if (versions.length === 0) return null;
 
  // Pick the highest version found
  versions.sort((a, b) => Bun.semver.order(a.slice(1), b.slice(1)));
  return versions[versions.length - 1];
}

A small updateCamkit function then writes that version straight into the course site's local data file, replacing the old version string, so the number updates itself the moment the monitor's next check runs, without a page refresh or manual edit. The full webhook server, including the file-writing step, is in this gist.

Page monitoring only works when you already know the URL. For something like AI model news, there's no single page that has everything. Swapping the target's type from "scrape" to "search" turns the monitor into an always-on web search instead of a page diff.

For example, the code below sets up a monitor that keeps an eye on the whole web for Claude Fable 5 news, checking every few hours and pinging you only when something genuinely new turns up:

const monitor = await fc.createMonitor({
  name: "Claude Fable 5 news",
  schedule: { text: "every 6 hours", timezone: "UTC" },
  targets: [
    {
      type: "search",
      queries: ["Claude Fable 5 update", "Claude Fable 5 release"],
      searchWindow: "24h",
      maxResults: 10,
    },
  ],
  goal: `Notify me about news, updates,
    or changes to Anthropic's Claude Fable 5 model`,
  webhook: {
    url: "https://fable-5-news.pages.dev/api/webhook",
    events: ["monitor.page"],
  },
  notification: {
    email: {
      enabled: true,
      recipients: ["you@example.com"],
      includeDiffs: true,
    },
  },
});

A search target swaps the page-level urls for two search-specific fields:

• queries is a list of 1 to 12 search terms the monitor runs on every check.
• searchWindow is a recency filter like "24h" or "7d" that scopes results to what's actually new.

Every check runs those queries, dedupes the results by canonical URL, and the AI judge scores each new result against the goal.

Notifications behave just like the page monitor, with two things worth calling out:

• webhook.events scopes which event types get posted (monitor.page fires per matched result, monitor.check.completed fires once per full check).
• email adds a second channel alongside the webhook, since a monitor can send both at once.

Only new, on-topic results trigger a notification, and that output can feed a site's news section just like the CamKit version number did earlier.

This exact monitor populates a live Fable 5 news page, where each entry was added automatically as the monitor caught it.

One endpoint covers a single known page, a whole website crawled on a schedule, or the entire web, all with the same scheduling, judging, and notification model underneath.

Check out the Firecrawl monitoring docs to go deeper.