Build a Finance Research Agent with Live Web Search + Scraping Using Firecrawl

Ask an LLM about a company's most recent earnings. It won't refuse. It will answer confidently, cite specific numbers, and mostly be wrong.

Not hallucinating wrong. Stale wrong. LLM training data has a cutoff. Financial data has a quarterly clock. The two don't align.

A finance research agent fixes this by removing the LLM from the loop on facts. It scrapes live sources first, then hands the data to the model for analysis. The model reasons. It doesn't recall.

In this guide, you'll build a finance research agent that scrapes live data from SEC EDGAR, company IR pages, and the web, then passes it to an LLM for analysis.

A finance research agent is a system that combines live web retrieval with an LLM to answer financial questions using current, sourced information. The key word is live. The agent doesn't recall financial data from training. It fetches it from the web each time it runs, extracts what's relevant, and passes it as context to the model.

This is fundamentally different from a finance chatbot. A chatbot draws on training data. A finance research agent draws on the real web at query time: SEC filings, company IR pages, financial news.

Firecrawl handles retrieval and extraction. Point it at a financial page and it returns clean markdown or structured JSON: both ready to pass directly to an LLM.

LLMs have a knowledge cutoff. For financial use cases, this is a hard constraint. A model trained on data through early 2025 has no knowledge of what happened in Q4 2025 earnings season. When you ask about a company's most recent quarter, the model either declines or, more dangerously, answers with data from a prior period. The distinction between training data vs live web data matters more in finance than almost any other domain.

@nicbstme on building AI agents for financial services.

A 2025 arXiv study evaluating 197,000+ financial questions found that LLMs display what researchers call "retrograde knowledge bias": they answer revenue questions correctly for 54% of companies when asked about 2017 data, but only 6% when asked about 1995 data. Accuracy degrades unpredictably across time periods, even for public data that has been available for decades.

The FailSafeQA benchmark found a 41% hallucination rate for financial queries under adversarial inputs. These are questions that are slightly ambiguous or incomplete, exactly the kind a real user would ask.

Institutional financial data is expensive.

@FundamentEdge on why finance is one of the hardest LLM verticals.

Bloomberg Terminal runs approximately $28,320–$31,980 per seat per year, according to third-party transaction records (as of May 2026). Bloomberg doesn't publish official pricing. FactSet starts around $4,000/user and scales to $30,000/user depending on modules. LSEG Workspace (formerly Refinitiv Eikon) runs $10,000–$42,000/user.

For developers building AI applications, this is a non-starter. As one developer put it on Hacker News, the major vendors are "predatory and will change the goalposts to charge you as much as you can bear."

r/algotrading: where do developers get financial data for backtesting?

yfinance is an open-source Python library that pulls financial data from Yahoo Finance. It's widely used for fetching historical prices, fundamentals, and earnings data with a simple API. The obvious workaround for developers, but it's not a real API. It scrapes Yahoo Finance's HTML. Yahoo tightened its rate limits around early 2024, and developers now report hitting YFRateLimitError after 4–5 requests per day. A tool that breaks at 5 requests/day isn't usable in any agent workflow.

SEC EDGAR is a public, government-operated database. Every 10-K, 10-Q, 8-K, and earnings release is available at no cost. Company IR pages are public. Financial news is public. The data is there. The challenge is extracting it reliably from pages that use JavaScript rendering or XBRL markup. That's a smart web search and scraping problem, not a data access problem.

That's exactly what Firecrawl solves.

The agent should use Search before scraping. Company IR page structures change frequently. SEC EDGAR filing URLs for specific filings are not guessable. A search step dynamically discovers the right current URL instead of relying on hardcoded paths that break.

r/algotrading: manually scraping 12 years of SEC EDGAR data.

Among these sources, company IR pages deserve special attention. They're often the fastest source for earnings data. A company publishes its earnings release on its IR page the moment it goes out, before any aggregator picks it up and days before the corresponding 10-Q lands on EDGAR. For time-sensitive analysis, this gap matters.

Apple's IR page, updated minutes after each earnings release. The SEC filing typically lands on EDGAR days later.

Most financial data sources are not friendly to standard scrapers. Here's what makes them hard, and how Firecrawl handles each issue.

The SEC EDGAR filing viewer uses dynamic rendering. Many company IR pages are React SPAs. A standard fetch + HTML parser returns empty or incomplete content.

What you get when you hit SEC.gov without JavaScript rendering. Firecrawl renders the page and returns the filing content.

Firecrawl renders JavaScript server-side. The same API call works on static HTML and JS-heavy SPAs. No configuration change needed.

Firecrawl strips navigation, footers, cookie banners, and ad chrome by default. What comes back is the document content, not the page wrapper. For a 10-K filing, this means the financial statements and MD&A text, not the 30% EDGAR navigation chrome surrounding them.

Every token of noise is a token wasted in the LLM context window. Clean input makes the analysis better and the cost lower.

Traditional scrapers require CSS selectors or XPaths. These break every time a site redesigns. Firecrawl's JSON extraction mode accepts a Zod schema. You describe the fields you want; Firecrawl locates them using LLM-assisted extraction. This works across different company IR page layouts without custom selectors per company.

Try it for yourself at firecrawl.dev/playground: paste any earnings release URL and see the clean markdown output.

Not all financial documents live on the web. Downloaded 10-K PDFs, emailed research reports, and exported spreadsheets are common inputs for finance workflows.

Firecrawl's /parse endpoint handles these directly. Upload a PDF, DOCX, or spreadsheet and get back the same clean markdown or structured JSON that /scrape returns for web pages. Tables come back with reading order preserved. You can request structured JSON extraction in the same call.

This means one consistent pipeline handles both live web retrieval and local file processing. No separate AI PDF parser needed.

The finance research agent follows four steps: Search, Scrape, Extract, Analyze. Each maps to a Firecrawl primitive or the LLM layer.

The Search API takes a natural-language query and returns URLs with page content. For a finance agent, the query might be:

"Apple FY26 Q2 earnings release investor relations"

The response includes the URL, title, and optionally the full page content in markdown. For simple queries, the search result already contains enough context. No second scrape needed.

Searching first means the agent always finds the current, correct source, even if company IR page URLs change between quarters. For a broader comparison of deep research APIs suited to agentic research workflows, that guide covers the tradeoffs.

The Scrape API takes a URL and returns clean markdown. onlyMainContent: true strips headers, footers, and nav. What remains is the document body.

For a 10-K filing, this is hundreds of pages. The agent should scrape the filing, then chunk by section headers before passing to the LLM. Firecrawl's markdown preserves heading structure, which makes section-based chunking straightforward.

When you need typed values (revenue, EPS, guidance), define a Zod schema and pass it to the scrape endpoint as a JSON format object. Firecrawl extracts those values and returns structured JSON. No selectors. No regex. The schema works across different IR page layouts.

The LLM receives the scraped markdown and extracted JSON as context. It's not being asked to recall financial data from training. It's being asked to reason over the data you just retrieved. This is the difference between asking a model to know something and asking it to think about something.

Always include the source URL and a retrieval timestamp in the prompt. The model can then cite its sources in the output.

Install dependencies:

npm install @mendable/firecrawl-js ai @ai-sdk/anthropic zod

Set environment variables:

FIRECRAWL_API_KEY=fc-...
ANTHROPIC_API_KEY=sk-ant-...

The agent is split into two files. tools.ts defines three Firecrawl-powered agent tools. agent.ts wires them into a generateText loop with a finance-specific system prompt.

Part 1: Define the tools (tools.ts)

import { Firecrawl, type SearchResultWeb } from '@mendable/firecrawl-js';
import { tool, zodSchema } from 'ai';
import { z } from 'zod';
 
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
 
const FinancialMetricsSchema = z.object({
  company_name: z.string(),
  reporting_period: z.string(),
  revenue_millions: z.number().optional(),
  net_income_millions: z.number().optional(),
  earnings_per_share: z.number().optional(),
  revenue_guidance_next_quarter: z.number().optional(),
  key_risks: z.array(z.string()).optional(),
});
 
export const searchFinancialSources = tool({
  description: 'Search the web for financial news, SEC filings, and earnings releases',
  inputSchema: z.object({
    query: z.string().describe('Search query, e.g. "Apple FY26 Q2 earnings investor relations"'),
    limit: z.number().optional().default(5),
  }),
  execute: async ({ query, limit }) => {
    const results = await firecrawl.search(query, { limit });
    const items = (results.web ?? []) as SearchResultWeb[];
    return items.map(r => ({
      url: r.url ?? '',
      title: r.title ?? '',
      content: r.description ?? '',
    }));
  },
});
 
export const scrapeFinancialPage = tool({
  description: 'Scrape a financial page (SEC filing, earnings release, IR page) and return clean markdown',
  inputSchema: z.object({
    url: z.string().url().describe('URL to scrape'),
  }),
  execute: async ({ url }) => {
    const result = await firecrawl.scrape(url, {
      formats: ['markdown'],
      onlyMainContent: true,
    });
    return { content: result.markdown?.slice(0, 10000) ?? '' };
  },
});
 
export const extractFinancialMetrics = tool({
  description: 'Extract structured financial metrics (revenue, EPS, guidance) as typed JSON from a page',
  inputSchema: z.object({
    url: z.string().url().describe('Earnings release or IR page URL'),
  }),
  execute: async ({ url }) => {
    const result = await firecrawl.scrape(url, {
      formats: [{ type: 'json', schema: zodSchema(FinancialMetricsSchema).jsonSchema as Record<string, unknown> }],
    });
    return result.json ?? {};
  },
});

Part 2: The agent (agent.ts)

import { generateText, stepCountIs } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { searchFinancialSources, scrapeFinancialPage, extractFinancialMetrics } from './tools';
 
export async function runFinanceAgent(company: string, question: string) {
  const { text } = await generateText({
    model: anthropic('claude-sonnet-4-6'),
    system: `You are a financial analyst with live web access via Firecrawl.
      
Use searchFinancialSources to find relevant sources first. Then scrapeFinancialPage to read the full content of the most relevant pages. Use extractFinancialMetrics when you need structured numbers (revenue, EPS, guidance). Always cite the source URL and retrieval date in your output.`,
    prompt: `Research ${company}: ${question}. Write a 3–5 paragraph research summary with citations.`,
    tools: { searchFinancialSources, scrapeFinancialPage, extractFinancialMetrics },
    stopWhen: stepCountIs(5),
  });
 
  return text;
}
 
const result = await runFinanceAgent('Apple', 'FY26 Q2 earnings and revenue guidance');
console.log(result);

Three things about this code:

Each tool has a single job. searchFinancialSources finds URLs. scrapeFinancialPage retrieves full page content as clean markdown. extractFinancialMetrics returns typed fields. The model decides which tool to call and in what order based on the system prompt and what it already knows. This is the core pattern behind AI agents.

The system prompt directs tool use. Rather than relying on implicit model behavior, the prompt explicitly tells the agent to search first, then scrape, then extract when structured data is needed. This ordering produces better results and costs less.

stopWhen: stepCountIs(5) manages the loop. The model calls tools in sequence without any manual orchestration. The loop runs until the model decides it has enough context or hits the step limit. No custom retry logic or state management needed.

Firecrawl's Search and Scrape workflow makes all of these buildable in a weekend. Get started at firecrawl.dev.

A finance research agent is not a replacement for a financial data API. Use a dedicated API (Alpha Vantage, Polygon.io, Tiingo) when you need:

• Tick-level or OHLCV price data: structured APIs provide this cleanly; web scraping is the wrong tool
• Normalized fundamentals at scale: querying 500 companies' P/E ratios is a database problem, not a scraping problem
• Real-time streaming data: WebSocket feeds from dedicated APIs are lower latency than scrape-on-demand

The finance research agent excels at qualitative and narrative data: what management said about margins, what changed in the risk factor section, how a competitor described its pricing strategy. These are questions no structured API answers. They're answered by the real web, and Firecrawl gets you there.

For more context on the LLM hallucination problem in financial applications, the FailSafeQA paper (arXiv:2502.06329) and the retrograde knowledge bias study (arXiv:2504.00042) are worth reading. Both establish why retrieval-augmented approaches outperform static LLM recall for financial queries.

For a complete look at how Firecrawl fits into investment and finance workflows, see the finance use cases documentation. For a breakdown of finance-first APIs worth combining with this approach, see best investment research APIs.

The agent in this guide is intentionally minimal. It runs on demand for a single company and question. From here, you can:

• Schedule it to run before earnings dates using a cron job (see automated data collection patterns for scheduling and ETL approaches)
• Add a vector store to chunk and index long 10-K filings for section-level retrieval
• Feed the output into a Slack notification or email digest
• Extend the tools to track SEC EDGAR's RSS feed for new 8-K filings

The Firecrawl Search and Scrape APIs are the consistent foundation: the data layer that handles the real web while your agent focuses on analysis.

Start for free at firecrawl.dev/pricing. The free tier includes 1,000 credits, enough to run several full agent sessions before you need to upgrade.