v1.6.0: LLMs.txt API and the MCP server

v1.6.0 Release

Introducing LLMs.txt API

The /llmstxt endpoint allows you to transform any website into clean, LLM-ready text files. Simply provide a URL, and Firecrawl will crawl the site and generate both llms.txt and llms-full.txt files that can be used for training or analysis with any LLM.

Docs here: https://docs.firecrawl.dev/features/alpha/llmstxt

Introducing Deep Research API (Alpha)

The /deep-research endpoint enables AI-powered deep research and analysis on any topic. Simply provide a research query, and Firecrawl will autonomously explore the web, gather relevant information, and synthesize findings into comprehensive insights.

Join the waitlist here: https://www.firecrawl.dev/deep-research

Official Firecrawl MCP Server

Introducing the Firecrawl MCP Server. Give Cursor, Windsurf, Claude enhanced web extraction capabilities. Big thanks to @vrknetha, @cawstudios for the initial implementation!

See here: https://github.com/firecrawl/firecrawl-mcp-server

Fixes & Enhancements

  • Improved charset detection and re-decoding.
  • Fixed extract token limit issues.
  • Addressed issues with includes/excludes handling.
  • Fixed AI SDK handling of JSON responses.

New Features & Improvements

  • AI-SDK Migration – transitioned to AI-SDK.
  • Auto-Recharge Emails – notify users about upgrades.
  • Fire-Index Added – introduced a new indexing system.
  • Self-Hosting Enhancements – OpenAI-compatible API & Ollama env support.
  • Batch Billing – streamlined billing processes.
  • Supabase Read Replica Routing – improved database performance.

Crawler & AI Improvements

  • Implemented Claude 3.7 and GPT-4.5 web crawlers.
  • Added Groq Web Crawler example.
  • Updated crawl-status behavior for better error handling.
  • Improved cross-origin redirect handling.

Documentation & Maintenance

  • Updated Dockerfile.
  • Fixed missing "required" field in docs.