Description
llms.txt Generator π
Generate consolidated text files from websites for LLM training and inference. Powered by @firecrawl_dev for web crawling and GPT-4-mini for text processing.
Features
- Crawls websites and combines content into a single text file
- Generates both standard (
llms.txt) and full (llms-full.txt) versions - Web interface and API access available
- No API key required for basic usage
Usage
Web Interface
Visit llmstxt.firecrawl.dev to generate files through the browser.
API Endpoint
GET https://llmstxt.firecrawl.dev/[YOUR_URL_HERE]
Note: Processing may take several minutes due to crawling and LLM operations.
Local Development
Prerequisites
Create a .env file with the following variables:
FIRECRAWL_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
OPENAI_API_KEY=
Installation
npm install
npm run dev
Related Templates
Explore more templates similar to this one
Top Italian Restaurants in SF
Search for websites that contain the top italian restaurants in SF. With page content
Quotes.toscrape.com Scrape
Zed.dev Crawl
The first step of many to create an LLM-friendly document for Zed's configuration.
Developers.campsite.com Crawl
o3 mini Company Researcher
This Python script integrates SerpAPI, OpenAI's O3 Mini model, and Firecrawl to create a comprehensive company research tool. The workflow begins by using SerpAPI to search for company information, then leverages the O3 Mini model to intelligently select the most relevant URLs from search results, and finally employs Firecrawl's extraction API to pull detailed information from those sources. The code includes robust error handling, polling mechanisms for extraction results, and clear formatting of the output, making it an efficient tool for gathering structured company information based on specific user objectives.