Mastering Firecrawl Search Endpoint: Web Search and Data Extraction in One API Call

curl -X POST https://api.firecrawl.dev/v2/search \
  -H "Authorization: Bearer fc-YOUR-KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "best project management tools", "limit": 5}'

from firecrawl import Firecrawl
app = Firecrawl()
results = app.search(query="best project management tools", limit=5)

npm install -g firecrawl-cli
firecrawl login --api-key fc-YOUR-KEY
firecrawl search "best project management tools" --limit 5

claude mcp add firecrawl --url https://mcp.firecrawl.dev/fc-YOUR-KEY/v2/mcp

pip install firecrawl-py python-dotenv

echo "FIRECRAWL_API_KEY='fc-YOUR-KEY-HERE'" >> .env

from firecrawl import Firecrawl
from dotenv import load_dotenv
 
load_dotenv()
app = Firecrawl()

results = app.search(query="best project management tools", limit=5)

print(f"Found {len(results.web)} web results")
print(f"Search completed successfully")

Found 5 web results
Search completed successfully

# First result details
first_result = results.web[0]
print(f"Title: {first_result.title}")
print(f"URL: {first_result.url}")
print(f"Description: {first_result.description}")
print(f"Category: {first_result.category}")

Title: 25 Best Project Management Software Picked For 2025
URL: https://thedigitalprojectmanager.com/tools/best-project-management-software/
Description: Explore top-rated project management software handpicked by experts to help you manage teams, timelines, and tasks with ease.
Category: None

All 5 results:
1. 25 Best Project Management Software Picked For 2025
   https://thedigitalprojectmanager.com/tools/best-project-management-software/
 
2. What is the best free project management tool, specifically geared ...
   https://www.reddit.com/r/projectmanagement/comments/1b0lfvi/what_is_the_best_free_project_management_tool/
 
3. Honest Review of 6 Personal Project Management Tools ... - ICAgile
   https://www.icagile.com/resources/honest-review-of-6-personal-project-management-tools-with-kanban-view
 
4. Manage your team's work, projects, & tasks online • Asana
   https://asana.com/
 
5. Project Management Software for Teams - Microsoft
   https://www.microsoft.com/en-us/microsoft-365/planner/project-management

try:
    results = app.search(query="python web scraping", limit=10)
    if results.web:
        print(f"Found {len(results.web)} results")
    else:
        print("No results found")
except Exception as e:
    print(f"Search failed: {e}")

results = app.search(query="", limit=5)

Empty query properly failed: Query cannot be empty

results = app.search(
    query="machine learning frameworks 2025",
    limit=3,
    sources=["web", "news"],
    timeout=30000  # 30 seconds
)

Advanced search found 3 results
1. 8 Best Machine Learning Software To Use in 2025 | Anaconda
2. AI Frameworks: Top Types To Adopt in 2025 - Splunk
3. Uses for ML frameworks like Pytorch/Tensorflow/etc in 2025 - Reddit

# This search finds pages, then scrapes each one automatically
content_results = app.search(
    query="web scraping best practices",
    limit=2,
    scrape_options={
        "formats": ["markdown", "links"]
    }
)
 
print(f"Content search found {len(content_results.web)} results")
 
for i, result in enumerate(content_results.web, 1):
    print(f"\nResult {i}:")
    print(f"Title: {result.metadata.title}")
    print(f"URL: {result.metadata.url}")
    print(f"Content length: {len(result.markdown)} characters")
    print(f"Found {len(result.links)} links in content")

Content search found 2 results
 
Result 1:
Title: Web Scraping Best Practices and Tools 2025 - ZenRows
URL: https://www.zenrows.com/blog/web-scraping-best-practices
Content length: 25234 characters
Found 51 links in content
 
Result 2:
Title: 7 Web Scraping Best Practices You Must Be Aware of
URL: https://research.aimultiple.com/web-scraping-best-practices/
Content length: 22384 characters
Found 111 links in content

results = app.search(
    query="python tutorials",
    limit=1,
    scrape_options={
        "formats": ["markdown", "html", "links"]
    }
)
 
# Each result contains full content in multiple formats
result = results.web[0]
print(f"Markdown: {len(result.markdown)} characters")
print(f"HTML: {len(result.html)} characters")
print(f"Links: {len(result.links)} found")

Markdown: 33640 characters
HTML: 221019 characters
Links: 302 found

results = app.search(
    query="machine learning research 2025",
    limit=2,
    scrape_options={"formats": ["markdown"]}
)
 
for result in results.web:
    # Content is already extracted and clean
    content = result.markdown
    word_count = len(content.split())
 
    print(f"Found article: {result.metadata.title}")
    print(f"Content: {word_count} words")

Found article: 2025 Conference
Content: 466 words
Found article: Apple Machine Learning Research at ICML 2025 - Apple Machine Learning Research
Content: 2070 words

def display_results(results, result_type="web", title="Results"):
    result_list = getattr(results, result_type, [])
    print(f"{title}:")
    for result in result_list:
        print(f"- {result.title}")
        print(f"  Category: {getattr(result, 'category', 'N/A')}")
        print(f"  URL: {result.url}")

github_results = app.search(
    query="python web scraping",
    limit=3,
    categories=["github"],
)
 
display_results(github_results, title="GitHub category results")

GitHub category results:
- Scrapy, a fast high-level web crawling & scraping ... - GitHub
  Category: github
  URL: https://github.com/scrapy/scrapy
- Web Scraping With Python - GitHub
  Category: github
  URL: https://github.com/KOrfanakis/Web_Scraping_With_Python
- awesome-web-scraping/python.md at master - GitHub
  Category: github
  URL: https://github.com/lorien/awesome-web-scraping/blob/master/python.md

research_results = app.search(
    query="machine learning transformers",
    limit=3,
    categories=["research"],
)
 
display_results(research_results, title="Research category results")

Research category results:
- [2304.10557] An Introduction to Transformers - arXiv
  Category: research
  URL: https://arxiv.org/abs/2304.10557
- Introduction to Sequence Modeling with Transformers - arXiv
  Category: research
  URL: https://arxiv.org/html/2502.19597v1
- Multimodal Learning With Transformers: A Survey - PubMed
  Category: research
  URL: https://pubmed.ncbi.nlm.nih.gov/37167049/

pdf_results = app.search(
    query="deep learning",
    limit=3,
    categories=["pdf"],
)
 
display_results(pdf_results, title="PDF category results")

PDF category results:
- [PDF] Deep Learning
  Category: pdf
  URL: https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf
- [PDF] The Principles of Deep Learning Theory - Purdue Engineering
  Category: pdf
  URL: https://engineering.purdue.edu/DeepLearn/Resources/DeepLearningTheory.pdf
- [PDF] Deep Learning - Microsoft
  Category: pdf
  URL: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/DeepLearning-NowPublishing-Vol7-SIG-039.pdf

# Helper function for source comparison
def compare_sources(query, limit=2):
    web_results = app.search(query=query, limit=limit, sources=["web"])
    news_results = app.search(query=query, limit=limit, sources=["news"])
 
    print("Web source results:")
    for result in web_results.web:
        domain = result.url.split('/')[2] if '/' in result.url else result.url
        print(f"- {result.title}")
        print(f"  Domain: {domain}")
 
    print("\nNews source results:")
    if hasattr(news_results, 'news') and news_results.news:
        for result in news_results.news:
            print(f"- {result.title}")
            print(f"  Date: {getattr(result, 'date', 'N/A')}")
 
compare_sources("AI developments 2025")

Web source results:
- The 2025 AI Index Report | Stanford HAI
  Domain: hai.stanford.edu
- The Latest AI News and AI Breakthroughs that Matter Most: 2025
  Domain: www.crescendo.ai
 
News source results:
- Microsoft's Unprecedented 2025: Cloud and AI Fuel Record Earnings, Igniting Shareholder Optimism
  Date: 12 hours ago
- 12 Graphs That Explain the State of AI in 2025
  Date: Apr 7, 2025

combined = app.search(
    query="AI developments 2025",
    limit=5,
    sources=["web", "news"]
)
 
print(f"Web: {len(combined.web)} results")
print(f"News: {len(combined.news) if hasattr(combined, 'news') else 0} results")

Web: 5 results
News: 5 results

# Basic image search
basic_images = app.search(
    query="mountain landscape",
    limit=3,
    sources=["images"]
)
 
print("Basic image search results:")
if hasattr(basic_images, 'images') and basic_images.images:
    print(f"Found {len(basic_images.images)} images")
    for i, img in enumerate(basic_images.images, 1):
        print(f"{i}. {img.title}")
        print(f"   Source page: {img.url}")
        print(f"   Position: {img.position}")

Basic image search results:
Found 3 images
1. Free Sunset Mountain Landscape Image | Download at StockCake
   Source page: https://stockcake.com/i/sunset-mountain-landscape_1219985_159231
   Position: 1
 
2. Watercolor Mountain Landscape Tutorial: Step-by-Step Guide
   Source page: https://www.esperoart.com/mountain-landscape-watercolor-tutorial/
   Position: 2
 
3. Mountain Landscape with Forests and a Stream · Free Stock Photo
   Source page: https://www.pexels.com/photo/mountain-landscape-with-forests-and-a-stream-18682254/
   Position: 3

def search_images_by_resolution():
    # Search for Full HD images
    hd_images = app.search(
        query="sunset landscape imagesize:1920x1080",
        limit=3,
        sources=["images"]
    )
 
    print("HD image search results:")
    if hasattr(hd_images, 'images') and hd_images.images:
        for i, img in enumerate(hd_images.images, 1):
            print(f"{i}. {img.title}")
            print(f"   Source: {img.url}")
 
    # Search for 4K images
    uhd_results = app.search(
        query="nature wallpaper imagesize:3840x2160",
        limit=2,
        sources=["images"]
    )
 
    print("\n4K image search results:")
    if hasattr(uhd_results, 'images') and uhd_results.images:
        for img in uhd_results.images:
            print(f"- {img.title}")
            print(f"  Source: {img.url}")
 
search_images_by_resolution()

HD image search results:
1. Stunning sunset landscape: Mediterranean rocks with olive trees in the sea.  Photorealistic motion background. 3D rendering.
   Source: https://www.storyblocks.com/video/stock/photorealistic-motion-background-mediteranean-rocks-in-the-sea-with-olive-trees-growing-on-them-beautiful-sunset-landscape-scene-3d-rendering-bymitreh8kaz2db71
 
2. Sunset in a hilly rural landscape on the edge of a forest, Timelapse video
   Source: https://www.storyblocks.com/video/stock/sunset-in-a-hilly-rural-landscape-on-the-edge-of-a-forest-timelapse-video-351768226
 
4K image search results:
- 50 Nature Wallpapers (All 4K, No watermarks) - Album on Imgur
  Source: https://in.pinterest.com/pin/1002543567023869780/
- My top 10 of all time - Nature and Art : r/wallpapers
  Source: https://www.reddit.com/r/wallpapers/comments/1g1cymc/my_top_10_of_all_time_nature_and_art/

def compare_content_freshness(base_query):
    recent_results = app.search(
        query=f"{base_query} trends",
        limit=3,
        tbs="qdr:m"  # Past month
    )
 
    all_time_results = app.search(
        query=f"{base_query} fundamentals",
        limit=3
    )
 
    print("Recent results (past month):")
    for result in recent_results.web:
        print(f"- {result.title}")
 
    print("\nAll-time results (fundamentals):")
    for result in all_time_results.web:
        print(f"- {result.title}")
 
compare_content_freshness("machine learning")

Recent results (past month):
- Top 13 Machine Learning Trends CTOs Need to Know in 2025
- The Evolution of AI and ML: Trends, Impact, and Future Insights
- The Ultimate List of Machine Learning Statistics for 2025 - Itransition
 
All-time results (fundamentals):
- Introduction to Machine Learning Concepts - Training - Microsoft Learn
- Machine Learning Crash Course - Google for Developers
- Machine Learning Tutorial - GeeksforGeeks

multi_category_results = app.search(
    query="neural networks",
    limit=4,
    categories=["github", "research"]
)
 
print("Combined GitHub + Research results:")
for result in multi_category_results.web:
    print(f"- {result.title}")
    print(f"  Category: {getattr(result, 'category', 'N/A')}")
    print(f"  URL: {result.url}")

Combined GitHub + Research results:
- Neural Networks | Journal | ScienceDirect.com by Elsevier
  Category: research
  URL: https://www.sciencedirect.com/journal/neural-networks
- Neural Network - an overview | ScienceDirect Topics
  Category: research
  URL: https://www.sciencedirect.com/topics/social-sciences/neural-network
- Neural Network - an overview
  Category: research
  URL: https://www.sciencedirect.com/topics/computer-science/neural-network
- Supervised learning in DNA neural networks
  Category: research
  URL: https://www.nature.com/articles/s41586-025-09479-w

echo "FIRECRAWL_API_KEY='fc-YOUR-FIRECRAWL-KEY'" >> .env
echo "OPENAI_API_KEY='sk-YOUR-OPENAI-KEY'" >> .env

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh
 
source ~/.bashrc  # or ~/.zshrc
 
# uv expects to work in a Python project directory with a pyproject.toml file
uv init
 
# Install project dependencies
uv add firecrawl-py python-dotenv langchain-openai langgraph

pip install firecrawl-py python-dotenv langchain-openai langgraph

#!/usr/bin/env python3
"""Research Assistant Agent using LangGraph and Firecrawl Search API"""
 
import os
from dotenv import load_dotenv
from firecrawl import Firecrawl
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import MemorySaver
 
# Load environment variables
load_dotenv()
firecrawl = Firecrawl()
 
def general_web_search(query: str, limit: int = 3) -> str:
    """Search the web for general information and extract full content."""
    try:
        results = firecrawl.search(
            query=query,
            limit=limit,
            sources=["web"],
            scrape_options={"formats": ["markdown"]}
        )
 
        if not results.web:
            return f"No web results found for: {query}"
 
        formatted_results = f"Web Search Results for '{query}':\n\n"
        for i, result in enumerate(results.web, 1):
            title = result.title
            url = result.url
            content_preview = result.markdown[:300] + "..." if len(result.markdown) > 300 else result.markdown
 
            formatted_results += f"{i}. {title}\n"
            formatted_results += f"   URL: {url}\n"
            formatted_results += f"   Content: {content_preview}\n\n"
 
        return formatted_results
 
    except Exception as e:
        return f"Error in web search: {str(e)}"

def academic_research_search(topic: str, limit: int = 3) -> str:
    """Search for academic research papers and scholarly content."""
    try:
        results = firecrawl.search(
            query=topic,
            limit=limit,
            categories=["research"],
            scrape_options={"formats": ["markdown"]}
        )
 
        if not results.web:
            return f"No research papers found for: {topic}"
 
        formatted_results = f"Academic Research Results for '{topic}':\n\n"
        for i, result in enumerate(results.web, 1):
            title = result.title
            url = result.url
            category = getattr(result, 'category', 'research')
 
            formatted_results += f"{i}. {title}\n"
            formatted_results += f"   Category: {category}\n"
            formatted_results += f"   URL: {url}\n\n"
 
        return formatted_results
 
    except Exception as e:
        return f"Error in research search: {str(e)}"
 
def code_search(technology: str, limit: int = 3) -> str:
    """Search GitHub repositories for code examples and projects."""
    try:
        results = firecrawl.search(
            query=technology,
            limit=limit,
            categories=["github"]
        )
 
        if not results.web:
            return f"No GitHub repositories found for: {technology}"
 
        formatted_results = f"GitHub Code Search Results for '{technology}':\n\n"
        for i, result in enumerate(results.web, 1):
            formatted_results += f"{i}. {result.title}\n"
            formatted_results += f"   URL: {result.url}\n"
            formatted_results += f"   Description: {result.description or 'No description available'}\n\n"
 
        return formatted_results
 
    except Exception as e:
        return f"Error in code search: {str(e)}"
 
def news_search(topic: str, limit: int = 3) -> str:
    """Search for recent news articles on a topic."""
    try:
        results = firecrawl.search(
            query=topic,
            limit=limit,
            sources=["news"],
        )
 
        news_results = []
        if hasattr(results, 'news') and results.news:
            news_results = results.news
        elif hasattr(results, 'web') and results.web:
            news_results = results.web
 
        if not news_results:
            return f"No recent news found for: {topic}"
 
        formatted_results = f"News Search Results for '{topic}':\n\n"
 
        for i, result in enumerate(news_results, 1):
            formatted_results += f"{i}. {result.title}\n"
            formatted_results += f"   URL: {result.url}\n"
            formatted_results += f"   Date: {getattr(result, 'date', 'Date not available')}\n\n"
 
        return formatted_results
 
    except Exception as e:
        return f"Error in news search: {str(e)}"

def create_research_agent():
    """Create the research assistant agent with search tools."""
 
    model = ChatOpenAI(model="gpt-5", temperature=0)
    tools = [general_web_search, academic_research_search, code_search, news_search]
 
    agent = create_react_agent(
        model=model,
        tools=tools,
        checkpointer=MemorySaver()
    )
 
    return agent

def main():
    """Main function to run the research agent."""
    print("🔍 Research Assistant Agent")
    print("=" * 50)
    print("I can help you search for:")
    print("• General web information with full content")
    print("• Academic research papers")
    print("• GitHub code repositories")
    print("• Recent news articles")
    print("\nType 'quit' to exit")
    print("=" * 50)
 
    agent = create_research_agent()
 
    while True:
        user_input = input("\n💬 What would you like to research? ")
 
        if user_input.lower() in ['quit', 'exit', 'q']:
            print("👋 Goodbye!")
            break
 
        if not user_input.strip():
            continue
 
        try:
            print("\n🤖 Searching...")
 
            system_message = """You are a Research Assistant Agent that helps users find information on any topic.
 
You have access to four specialized search tools:
1. general_web_search - For web search with full content extraction
2. academic_research_search - For finding academic papers and scholarly content
3. code_search - For finding GitHub repositories and code examples
4. news_search - For finding recent news articles
 
Choose the appropriate tool(s) based on the user's request. You can use multiple tools for extensive research."""
 
            response = agent.invoke(
                {"messages": [("system", system_message), ("user", user_input)]},
                config={"configurable": {"thread_id": "research_session"}}
            )
 
            final_message = response["messages"][-1]
            print(f"\n📋 Research Results:\n{final_message.content}")
 
        except Exception as e:
            print(f"\n❌ Error: {str(e)}")
 
if __name__ == "__main__":
    main()

python research_agent.py

🔍 Research Assistant Agent
==================================================
I can help you search for:
• General web information with full content
• Academic research papers
• GitHub code repositories
• Recent news articles
 
Type 'quit' to exit
==================================================
 
💬 What would you like to research?

🤖 Searching...
 
📋 Research Results:
Here are some recent machine learning frameworks for computer vision found on GitHub:
 
1. OpenCV: An open-source library providing computer vision and machine learning
   algorithms for image and video analysis. It also includes Detectron2, a
   high-performance object detection framework.
 
2. 500 AI Machine learning Deep learning Computer vision NLP Projects with code:
   This repository contains a list of 500+ AI projects, including machine learning
   and computer vision, with code.
 
3. Lightly: A computer vision framework for self-supervised learning. The repository
   also includes Learnergy, energy-based machine learning models built upon PyTorch.

🤖 Searching...
 
📋 Research Results:
Here are some academic research papers on neural networks:
 
1. Neural Networks Journal | ScienceDirect: The official journal for neural
   network research with peer-reviewed articles.
 
2. Neural Network - an overview | ScienceDirect Topics: Comprehensive overview
   of neural networks in the context of social sciences.
 
3. Overview of artificial neural networks - PubMed: Medical and biological
   applications of artificial neural networks.

🤖 Searching...
 
📋 Research Results:
Here are some Python web scraping libraries and projects on GitHub:
 
1. Web Scraping With Python: Tutorial using requests and Beautiful Soup libraries.
 
2. Python-Web-Scraping-Tutorial by oxylabs: Comprehensive tutorial from simple
   examples to complex tasks.
 
3. Scrapling by D4Vinci: Powerful, flexible Python library for
   web scraping.
 
4. Scrapegraph-ai: Web scraping python library that uses LLM and direct graph
   logic to create scraping pipelines.