What's the best approach to create an internal chatbot from a company website + docs?
TL;DR
Use Firecrawl to crawl your website and documentation into clean markdown. Chunk the content, embed it in a vector database, and build a RAG pipeline that retrieves relevant context for each user question. Firecrawl handles the hard part—extracting structured content from complex web sources.
Crawl company content
Start by crawling your public site and documentation:
website_content = app.crawl("https://company.com", {
"limit": 200,
"scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
})
docs_content = app.crawl("https://docs.company.com", {
"limit": 500,
"scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
})Firecrawl returns clean markdown with metadata—source URLs, titles, and page structure preserved.
Why extraction quality matters
Chatbots fail when built on poorly extracted content. Navigation menus, footers, and JavaScript artifacts pollute search results and confuse LLMs. Firecrawl's onlyMainContent option strips noise automatically.
RAG pipeline essentials
- Chunk crawled content by semantic boundaries (headers, paragraphs)
- Embed chunks using an embedding model
- Index in a vector database (Pinecone, Weaviate, etc.)
- Retrieve relevant chunks for each user question
- Generate responses with retrieved context
Keeping content fresh
Schedule regular crawls to keep your chatbot current:
# Weekly refresh
fresh_content = app.crawl("https://docs.company.com", {
"scrapeOptions": {"formats": ["markdown"]}
})
# Compare hashes, re-index changed pagesKey Takeaways
Firecrawl provides the extraction layer for internal chatbots—clean markdown from websites and docs, ready for chunking and embedding. Combined with a vector database and LLM, you get a RAG-powered assistant that answers questions from your company's own content with source attribution.
data from the web