How to Create an llms.txt File for Any Website

Creating an llms.txt file for your website is now simpler than ever with the llms.txt Generator. This tool enables you to compile your entire website into a single text file that can be used with any Large Language Model (LLM), improving how AI interacts with your content.

An llms.txt file is a standardized markdown file proposed by Jeremy Howard to provide information to help LLMs use a website at inference time. Unlike traditional web content designed for human readers, llms.txt files offer concise, structured information that LLMs can quickly ingest. This is particularly useful for enhancing development environments, providing documentation for programming libraries, and offering structured overviews for various domains such as corporate websites, educational institutions, and personal portfolios.

The llms.txt file is located at the root path /llms.txt of a website and contains sections in a specific order, including a project name, a summary, detailed information, and file lists with URLs for further details. This format allows LLMs to efficiently access and process the most important information about a website.

The llms.txt Generator leverages Firecrawl to crawl your website and extracts data using gpt-4o-mini. You can generate both llms.txt and llms-full.txt files through the web interface or via API.

You can access llms.txt directly by making a GET request to:

http://llmstxt.firecrawl.dev/{YOUR_URL}

For the full version, use:

http://llmstxt.firecrawl.dev/{YOUR_URL}/full

If you have a Firecrawl API key, you can include it to unlock full results and remove limits:

http://llmstxt.firecrawl.dev/{YOUR_URL}?FIRECRAWL_API_KEY=YOUR_API_KEY

For the full version with API key:

http://llmstxt.firecrawl.dev/{YOUR_URL}/full?FIRECRAWL_API_KEY=YOUR_API_KEY

1. Visit the Generator: Go to http://llmstxt.firecrawl.dev.

2. Enter Your Website URL: Input the URL of your website.

3. Generate the File: Click the generate button and wait a few minutes as the tool processes your site.

4. Download Your Files: Once ready, download the llms.txt and llms-full.txt files.

While an API key is not required, using a free Firecrawl API key removes any usage limits and provides full access to all features.

• llms.txt Standard: Discover more about the llms.txt standard at http://llmstxt.org.

• Directory of Adopters: Check out the directory of llms.txt adopters at http://directory.llmstxt.cloud.

• Contribute: The llms.txt Generator is open source. Visit the GitHub repository to contribute: https://github.com/mendableai/llmstxt-generator.

• Firecrawl Official Website
• llms.txt Standard
• llms.txt Generator
• Directory of llms.txt Adopters
• llms.txt Generator GitHub Repository