--- type: "firecrawl-provider" description: "Public Substack archives, search and free posts or paid previews." use_when: "Public Substack archives, search and free posts or paid previews." categories: "News" capabilities: 3 credits_per_call: 5 --- # Substack on Firecrawl Alexandria Public Substack archives, search and free posts or paid previews. - Categories: News - Category index: [News category](https://firecrawl.dev/alexandria/agents/categories/news) - Provider key: `substack-com` - Access: Firecrawl credits - Cost: 5 credits per call - Delivery: First 100 words of each matching article ## More - [Human guide](https://firecrawl.dev/app/alexandria/substack-com) - [OpenAPI spec](https://firecrawl.dev/alexandria/agents/providers/substack-com/openapi.json) ## Capabilities - [Archive](https://firecrawl.dev/alexandria/agents/providers/substack-com/newsletters/archive): One page of a Substack publication's archive, newest, top or pinned first. Each entry carries title, subtitle, URL, publish time, audience (free or paid), word count, engagement counts, section, tags and bylines. Page with offset and limit. Custom domains are limited to the declared publications; paid text remains the anonymous preview. - [Post](https://firecrawl.dev/alexandria/agents/providers/substack-com/newsletters/post): One Substack post by URL, or by publication and slug: metadata plus body_html and body_text. Free posts (audience everyone) are complete; paid posts return the publicly served preview with paywalled true, full_text_available false and text_coverage showing how much of the author's word count is visible. Custom domains are limited to the declared publications; paid text remains the anonymous preview. - [Search](https://firecrawl.dev/alexandria/agents/providers/substack-com/newsletters/search): Search a Substack publication's archive for posts matching a query, using Substack's own archive search. Returns the same post entries as archive, with the query echoed. Custom domains are limited to the declared publications; paid text remains the anonymous preview. ## 1. Choose this provider when Public Substack archives, search and free posts or paid previews. ## Article preview policy News providers return metadata plus the first 100 words of each matching article. They are not wrapped third-party APIs and do not return full article text. ## 2. Minimal request Call `POST https://api.firecrawl.dev/v2/scrape` with `{ alexandria: { provider, capability, options } }`. For a batch, send `{ alexandria: [...] }` with up to 10 calls. ```json { "provider": "substack-com", "capability": "newsletters/archive", "options": { "limit": 1, "publication": "www.astralcodexten.com" } } ``` ## 3. Add provider options Use only the options needed for the task: - `limit` (number): limit Example: `12` - `offset` (number): offset Example: `0` - `publication` (string, required): Substack slug or a declared custom publication host (www.astralcodexten.com, www.lennysnewsletter.com, www.noahpinion.blog). Other custom domains are not supported by this program. Example: `` - `sort` (string): new = newest first, top = most popular, pinned = editor pins. Example: `new` ## 4. Request through your preferred interface ### JavaScript ```javascript const result = await firecrawl.scrape({ alexandria: { provider: "substack-com", capability: "newsletters/archive", options: { limit: 1, publication: "www.astralcodexten.com", }, }, }); ``` ### Python ```python result = firecrawl.scrape_alexandria({ "provider": "substack-com", "capability": "newsletters/archive", "options": { "limit": 1, "publication": "www.astralcodexten.com" } }) ``` ### cURL ```sh curl https://api.firecrawl.dev/v2/scrape \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "alexandria": { "provider": "substack-com", "capability": "newsletters/archive", "options": { "limit": 1, "publication": "www.astralcodexten.com" } } }' ``` ### CLI ```sh firecrawl scrape 'substack-com/newsletters/archive' \ --options '{"limit":1,"publication":"www.astralcodexten.com"}' ``` ### MCP Call the FCX MCP retrieve tool with this object: ```json { "provider": "substack-com", "capability": "newsletters/archive", "options": { "limit": 1, "publication": "www.astralcodexten.com" } } ``` Ask for only the returned fields needed by the task. ## 5. Full request shape ```json { "provider": "substack-com", "capability": "newsletters/archive", "options": { "limit": 1, "publication": "www.astralcodexten.com" } } ``` ## 6. Response data The response includes `success`, `provider`, `capability`, `creditsCost` and `data`. This example shows the provider payload in `data`: ```json { "count": 1, "has_more": true, "limit": 1, "next_offset": 1, "observed_at_ms": 1789435777564, "offset": 0, "paywalled_count": 0, "posts": [ { "audience": "everyone", "authors": [ { "bio": null, "handle": "astralcodexten", "id": "12009663", "name": "Scott Alexander", "photo_url": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7b500d22-1176-42ad-afaa-5d72bc36a809_44x44.png" } ], "comments": 35, "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/017c9c94-ad3d-490e-90b4-1306c0e97122_575x339.png", "description": "...", "free_preview_sent": false, "id": "214878061", "paywalled": false, "podcast_duration_seconds": null, "podcast_url": null, "preview_text": null, "publication_id": "89120", "published_at": "2026-09-14T23:56:40.626Z", "reactions": 94, "restacks": 2, "section": null, "slug": "king-ludd", "subtitle": "...", "tags": [], "title": "King Ludd", "type": "newsletter", "url": "https://www.astralcodexten.com/p/king-ludd", "wordcount": 2755 } ], "publication": { "host": "www.astralcodexten.com", "url": "https://www.astralcodexten.com" }, "query": null, "sort": "new", "source": "https://www.astralcodexten.com/api/v1/archive?sort=new&search=&offset=0&limit=1" } ``` ## API reference-derived contract This provider returns article metadata and a 100-word preview; use the source URL for the complete article. ### Archive - Capability: `newsletters/archive` - Description: One page of a Substack publication's archive, newest, top or pinned first. Each entry carries title, subtitle, URL, publish time, audience (free or paid), word count, engagement counts, section, tags and bylines. Page with offset and limit. Custom domains are limited to the declared publications; paid text remains the anonymous preview. - Instructions: One page of a Substack publication's archive, newest, top or pinned first. Each entry carries title, subtitle, URL, publish time, audience (free or paid), word count, engagement counts, section, tags and bylines. Page with offset and limit. - Cost: 5 credits per call - Capability file: [Archive](https://firecrawl.dev/alexandria/agents/providers/substack-com/newsletters/archive) Accepted options: - `limit` (number): limit Example: `12` - `offset` (number): offset Example: `0` - `publication` (string, required): Substack slug or a declared custom publication host (www.astralcodexten.com, www.lennysnewsletter.com, www.noahpinion.blog). Other custom domains are not supported by this program. Example: `` - `sort` (string): new = newest first, top = most popular, pinned = editor pins. Example: `new` Response schema example: ```json { "count": 1, "has_more": true, "limit": 1, "next_offset": 1, "observed_at_ms": 1789435777564, "offset": 0, "paywalled_count": 0, "posts": [ { "audience": "everyone", "authors": [ { "bio": null, "handle": "astralcodexten", "id": "12009663", "name": "Scott Alexander", "photo_url": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7b500d22-1176-42ad-afaa-5d72bc36a809_44x44.png" } ], "comments": 35, "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/017c9c94-ad3d-490e-90b4-1306c0e97122_575x339.png", "description": "...", "free_preview_sent": false, "id": "214878061", "paywalled": false, "podcast_duration_seconds": null, "podcast_url": null, "preview_text": null, "publication_id": "89120", "published_at": "2026-09-14T23:56:40.626Z", "reactions": 94, "restacks": 2, "section": null, "slug": "king-ludd", "subtitle": "...", "tags": [], "title": "King Ludd", "type": "newsletter", "url": "https://www.astralcodexten.com/p/king-ludd", "wordcount": 2755 } ], "publication": { "host": "www.astralcodexten.com", "url": "https://www.astralcodexten.com" }, "query": null, "sort": "new", "source": "https://www.astralcodexten.com/api/v1/archive?sort=new&search=&offset=0&limit=1" } ``` ### Post - Capability: `newsletters/post` - Description: One Substack post by URL, or by publication and slug: metadata plus body_html and body_text. Free posts (audience everyone) are complete; paid posts return the publicly served preview with paywalled true, full_text_available false and text_coverage showing how much of the author's word count is visible. Custom domains are limited to the declared publications; paid text remains the anonymous preview. - Instructions: One Substack post by URL, or by publication and slug: metadata plus body_html and body_text. Free posts (audience everyone) are complete; paid posts return the publicly served preview with paywalled true, full_text_available false and text_coverage showing how much of the author's word count is visible. - Cost: 5 credits per call - Capability file: [Post](https://firecrawl.dev/alexandria/agents/providers/substack-com/newsletters/post) Accepted options: - `publication` (string): Substack slug or a declared custom publication host (www.astralcodexten.com, www.lennysnewsletter.com, www.noahpinion.blog). Other custom domains are not supported by this program. Example: `` - `slug` (string): Pattern: ^[A-Za-z0-9._-]+$. Example: `` - `url` (string): https:///p/ or https://open.substack.com/pub//p/. Provide url, or publication and slug. Pattern: ^https?://. Example: `` Response schema example: ```json { "access_note": "Free post: body_text is the complete public article.", "audience": "everyone", "authors": [ { "bio": null, "handle": "astralcodexten", "id": "12009663", "name": "Scott Alexander", "photo_url": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7b500d22-1176-42ad-afaa-5d72bc36a809_44x44.png" } ], "body_html": "

Thanks to everyone who commented last week with prediction markets I missed. Two of them seemed to be especially interesting.

Polymarket is another cryptocurrency-based prediction market. It's got about two dozen contracts open, and some of them are pretty big - $5 million plus! With that kind of money, we ought to be seeing some really good predicting! We're...not. Either there's a 6% chance that Donald Trump will be president again by March 31, or something's gone wrong.

Probably it's the second one. I tried to bet against Trump, but getting money into the market was pretty hard. You need USDCoins, a stablecoin related to Ethereum. Polymarket tries to let you buy them directly, but their app wanted me to give them a security code which never showed up, so I gave up on this. Instead I bought some USDC at Coinbase and tried to send them over. But along with the usual Ethereum gas fees, they have something called a relayer, which is supposed to collect my money and put it in my account. And it's apparently heavily backed up, and after two days my money is nowhere to be seen (though I believe them when they say that they're trying their hardest and it will probably percolate through the Ethereum network someday). Maybe everyone's having these kinds of issues and this is why the Trump contract hasn't adjusted? I'm not sure. I will keep you updated if my money ever materializes.

Kalshi is \"the first regulated futures exchange dedicated to trading on event outcomes\". As far as I can tell they actually did it. They got the government to agree to let them run a prediction market, regulated as investing rather than gambling. There will be a $25,000 max on contracts, so it might not be enough for the real whales, but that's still thirty times higher than existing prediction markets and enough to probably be a sea change. I talked to a representative who said they'll be \"focused on creating markets in a few key categories such as climate, economics, geopolitics, energy, education, government, space, COVID, and technology\", but that they remain open to potentially expanding to all sorts of other areas (except onions, which apparently have a very specific carve-out as something which it is illegal to have futures markets about, somebody please write a cyberpunk novel revolving around this). So far they're still setting things up, but they describe themselves as \"ramping quickly towards launch\". I am super psyched about this and will keep you all updated.

Getting back to Metaculus, let’s look at what they’ve got on AI: