How AI Agents Can Reliably Search Academic Papers in 2026: 5 Academic Search APIs Compared

If you are building an AI agent that answers questions from academic literature, the academic search API you pick is usually what decides whether the answer holds up. An agent can reason well and still cite a paper that does not support the claim it was attached to.

That happens when the retrieval layer gives it an abstract and a title, then the model fills in evidence it never actually saw.

That failure has a few different shapes: unsupported attribution, a hallucinated identifier, a paraphrase that drifts from what the paper actually reported. The common thread is that the agent was asked to describe evidence it never saw.

Most tool comparisons in this space are written for researchers choosing something to open in a browser. Firecrawl's roundup of AI research tools covers that well, comparing Perplexity, Consensus, and Elicit. This guide looks at a different problem: which APIs you can wire into an agent workflow, and what each one does when you call it in a loop.

I checked every rate limit, benchmark number, and corpus figure below against a primary source in August 2026. Several of them changed this year, and a few are commonly cited wrong. Where two official sources disagree, I say so.

Academic search for an AI agent is a retrieval layer that gives the agent access to stable paper identity, supporting evidence from the paper itself, and the relationships between papers. It differs from academic search built for people, which optimizes for a human skimming a results page.

An agent cannot skim, so whatever the retrieval layer returns becomes the evidence the answer is built on.

In practice that means the retrieval layer needs to provide three things:

• Stable paper identity. A canonical identifier, title, authors, and date, as fields rather than text to parse.
• Relevant supporting content. The passage from the body that addresses the question, not just the abstract.
• Relationships to related work. A way to move from one paper to what it cites, what cites it, and what sits near it.

They do not have to arrive in one call. A Firecrawl Research Index workflow can use separate search, read, and related-paper operations. What matters is that the agent can get identity, evidence, and relationships without having to reconstruct the academic layer itself.

Plenty of useful tools provide one or two. A search endpoint returning ranked titles and abstracts is good at discovery, and semantic search APIs do this well, but the agent still cannot quote what it found.

A general scraper gives you page content and the source URL, which is enough for provenance. What it does not do is normalize the academic-specific parts: DOI, arXiv ID, PMID, paper version, structured author and publication metadata, canonical identity for the same paper across sources, and citation relationships.

You can rebuild those yourself, and for a one-off that is reasonable. Inside an agent loop it becomes work you repeat on every result.

Pairing identity with supporting content is what makes a citation verifiable, which is the practical goal behind grounding an LLM.

Academic search interfaces were designed around a person clicking through results: type a query, skim ten titles, open the promising one. Agent workflows put pressure on five specific things, and the same constraints apply to search tooling for agents generally.

That last row is where the published numbers moved most in 2026, and it is worth checking before you commit to a provider.

These are grouped by the workflow each one fits rather than ranked. No benchmark has been run across all five, so a single ordering would not be supportable.

A retrieval index built around agent workflows in AI/ML, rather than a bibliographic database with an API in front of it.

The Firecrawl Research Index launched in June 2026 covering 3M+ arXiv papers. It is one surface of Firecrawl's broader context API for AI agents, which also covers live web search, scraping pages into clean Markdown, and document parsing.

For academic work, the relevant pieces are the Firecrawl Research Index for papers and general Search for material discussed on the web but not published as a paper.

Papers and code are separate surfaces here. GitHub issues, merged pull requests, and READMEs live in the Developer Index, which ranks them with matched passages and adds curated documentation sources. So an agent that needs both a paper and its implementation makes two calls against two indexes.

On arXivQA, Firecrawl reports 53.3% recall at $0.32 per task, against 45.4% for the next best provider tested. It also reports an MRR of 0.750, which Firecrawl describes as placing the correct paper in the top two results. Firecrawl is the only provider in this comparison with a published arXivQA result, though see the benchmark caveat below.

Capabilities:

• GET /v2/search/research/papers : natural-language search over paper metadata and abstracts, with authors, categories, from, and to filters
• GET /v2/search/research/papers/{id} : metadata by default; add a query parameter and it switches to read mode, returning scored in-body passages
• GET /v2/search/research/papers/{id}/similar : citation-graph expansion in three modes, similar (co-citation and bibliographic coupling), citers, and references

For the code behind a paper, the documented path is the Developer Index: GET or POST /v2/search/developer, which returns doc, issue, pull_request, and readme results with matched passages and stable IDs like issue:owner/repo#123.

Access: No API key required to start for either index. Adding a key raises your rate limit.

Honest take: Read mode is the capability that matters most in practice. Asking a paper a question and getting scored passages from the body, instead of a PDF link you still have to parse, means the agent can quote something it actually retrieved.

When you do have a PDF in hand, whether it is a paper the index does not cover or a preprint an author sent you, Firecrawl's /parse endpoint turns it into clean Markdown in one call. That covers the same last-mile problem the arXiv API leaves open, since arXiv returns a PDF link and expects you to handle the extraction yourself.

Pairing that with a Developer Index lookup is useful when the question is how a method was implemented rather than what it claims. The tradeoff is that papers and implementation evidence now come from separate calls.

Cons: The AI/ML corpus does not cover economics or journals outside arXiv, and pricing is credit-based, which is harder to forecast than flat per-request pricing until you have measured your own traffic.

Firecrawl has since expanded coverage with a dedicated life sciences category spanning 41M+ drug discovery, clinical trial, and biology papers, so biomedical questions now have a first-party primary source.

The arXivQA result is also vendor-run. The methodology is published and reproducible, but Firecrawl ran it on its own index rather than through an independent evaluation.

The research endpoints are keyless to start, so you can run a query against the Firecrawl Research Index documentation before wiring anything into your agent.

Aemon (YC W26), which builds autonomous AI research engineers, saw the same pattern in their own benchmark of scientific and technical retrieval systems:

Aemon is building autonomous AI research engineers that solve hard scientific and technical problems. To do that, our systems must continuously learn from the frontier of research: papers, implementations, benchmarks, and technical discussions across the web.

We use Firecrawl Research as part of the retrieval stack behind Aemon. In our internal benchmark of scientific and technical retrieval systems, it delivered the strongest recall of any provider we tested, particularly at deeper search depths where comprehensive coverage is critical. Firecrawl consistently surfaced relevant scientific and technical sources that would otherwise have been missed.

-- Ray Xu, Co-Founder, Aemon (YC W26)

Free, stable, and well suited to direct lookups.

If you know which paper you want and it is on arXiv, this is the most direct way to get it. No key, no billing. For freshness-sensitive agents, arXiv posts scheduled announcements Sunday through Thursday, with none on Friday or Saturday. A submission received Friday morning is not scheduled for announcement before Sunday evening, and moderation can delay it further.

Capabilities:

• GET export.arxiv.org/api/query : search by field, sort by relevance or lastUpdatedDate, page with start and max_results

Access: Free, no authentication. The terms of use specify no more than one request every three seconds, limited to a single connection at a time.

Honest take: It works well as a resolver. Given an arXiv ID it returns clean, structured metadata, indefinitely, for free, and most stacks benefit from keeping it around for exactly that.

Cons: One request every three seconds with no concurrency works for occasional lookups, but it becomes restrictive in multi-step agent workflows. Twenty sequential calls cost a minute of wall-clock time before reasoning starts.

The API returns an abstract and a PDF link, so full text means fetching and parsing the PDF yourself. There is no citation graph, so the traversal pattern described later needs a different provider.

Broad cross-domain coverage with a genuinely good citation graph.

The Academic Graph API covers 214 million papers, 2.49 billion citations, and 79 million authors across every discipline, which makes it a practical default for anything spanning multiple fields.

Capabilities:

• /graph/v1/paper/search : keyword and relevance search across the corpus
• /graph/v1/paper/{id}/citations and /references : forward and backward traversal
• /graph/v1/snippet/search : query-matched text snippets, scored, with snippetKind and section so you know which part of the paper a snippet came from. Filterable by paperIds (up to roughly 100), authors, and fieldsOfStudy, with a default limit of 10 and a maximum of 1,000
• Batch endpoints accepting up to 500 paper IDs per call

The snippet endpoint is worth calling out because Semantic Scholar is often described as an abstracts-only API, and that is not accurate. Snippets are extracted from open-access papers or abstracts, and each result carries openAccessInfo with the license and source details.

That means an agent can get evidence from inside a paper here, not just a citation to one.

Two things it is not. It is not arbitrary full-document retrieval: you get ranked snippets, not the paper. Coverage is also bounded by open access, so for a paywalled paper the snippet may come from the abstract.

Firecrawl's read mode differs in shape rather than only in quality. You name a paper and get passages ranked against your question, while Semantic Scholar ranks snippets across the corpus. Constraining snippet search with paperIds gets you closer to the same operation.

Access: Free, and the rate limit works differently than most developers expect.

Unauthenticated requests share 1,000 requests per second across all anonymous users. That sounds generous until you apply it to a production agent: you are sharing that pool with everyone else who did not get a key, so throughput varies with global traffic and you cannot plan against it.

An API key gives you a dedicated 1 request per second across all endpoints, raised only after a review. That is predictable, but it becomes restrictive for bursty multi-hop retrieval.

If citation traversal is central to your workflow, use the batch endpoints, which let you resolve up to 500 IDs in one call. Request a higher limit through their partner form before the default becomes a production bottleneck.

On freshness, the monthly figure Semantic Scholar publishes covers its bulk S2AG datasets. That is the cadence for the downloadable corpus, not a statement about how quickly a new paper reaches the live API, which they do not publish.

Honest take: Semantic Scholar is a strong free option when citation traversal and cross-domain coverage matter. Between the graph endpoints, snippet search, and batch lookups, an agent can find a paper, pull supporting text, and walk the citations without leaving the API.

Cons: Snippet coverage is bounded by open access, so a paywalled paper may only yield an abstract-derived snippet. The rate limit structure is a tradeoff in both directions rather than a clear upgrade path. And no arXivQA result is published, so retrieval quality relative to Firecrawl is untested rather than settled.

A broad cross-domain metadata layer with a separate full-text content surface.

OpenAlex is a comprehensive index of research across disciplines. It is especially useful when an agent needs DOI resolution, structured author or institution metadata, references, or broad discovery outside a single research field.

Access: OpenAlex's current API pricing is dollar-denominated. Without a key, the API includes $0.10 of usage per day; a free key raises that to $1 per day.

Singleton lookups are free, list and filter calls cost $0.10 per 1,000, keyword search costs $1 per 1,000, and content downloads cost $10 per 1,000 files. The overall ceiling is 100 requests per second.

That free-key budget is roughly enough for 10,000 list or filter calls, 1,000 keyword searches, or 100 content downloads per day, depending on the mix. For an agent, that is more useful than treating every request as if it costs the same.

OpenAlex also caches full-text content (PDFs and machine-readable TEI XML) for a substantial subset of the corpus. That gives an agent access to the underlying document for many works.

It is still different from query-ranked passage retrieval. You need to parse or search the document and select the evidence relevant to the question.

For bulk data, the free public snapshot is updated quarterly. OpenAlex recommends the REST API for real-time application needs, while paid plans can access daily-refreshed snapshots.

Honest take: OpenAlex is a strong fit for cross-domain metadata, DOI resolution, citation relationships, and document acquisition when cached content exists. It complements a passage-retrieval layer well because it solves identity and breadth without pretending those are the same problem as evidence selection.

Cons: It does not return question-specific ranked passages. Downloading a PDF or TEI file gives the agent the document, not the answer-bearing evidence inside it, so grounded workflows still need a parsing and passage-selection step.

A web-wide neural search API that recently added an academic vertical.

Exa is a semantic search engine over the live web that returns full page content. That makes it useful for the material surrounding papers: lab blog posts, workshop pages, release notes, and talks that reference research without being research.

As of July 23, 2026, Exa offers a dedicated publications index. Its publications launch post says Exa searches that index alongside its broader web index, then combines and reranks the results. On its own publications evaluation, Exa reports 86.4% recall and 0.726 MRR.

Those numbers are not directly comparable to Firecrawl's 53.3%. Firecrawl's figure comes from arXivQA, a labeled arXiv retrieval benchmark. Exa's comes from its own publications benchmark on a different retrieval system and query set. The corpora and evaluation methods differ, and neither vendor has published a result on the other's benchmark.

Access: Exa documents a rate limit of 10 QPS on /search, with custom rates available on higher tiers. Search requests currently cost $7 per 1,000 requests for up to 10 results, with additional results billed separately. The /contents endpoint starts at $1 per 1,000 pages per content type. The public maximum is 100 results per search.

Honest take: Exa is a strong fit when the agent needs grey literature alongside papers, and flat per-request pricing is straightforward to forecast. Search can return full page content or targeted highlights, which can save a separate fetch step when the agent needs evidence from the surrounding web.

Cons: No citation graph, so forward and backward traversal needs another provider. Web results are URL-oriented rather than normalized around one canonical paper identity, so deduplication can still be your responsibility. Its academic recall figure is also self-reported on a vendor-designed evaluation, the same broad caveat that applies to Firecrawl's benchmark.

arXivQA comes from alphaXiv. Each query is labeled with up to 10 ground-truth arXiv IDs: papers that contain information answering the question. Firecrawl benchmarked on roughly 200 of these queries.

alphaXiv explains why they chose recall as the primary metric:

Documents from the retrieval agent would be returned to a top-level agent which then uses them to generate a final answer for the user. In this setting, it's much more damaging to miss a critical document than include a few unnecessary ones.

Recall maps well to this kind of workflow because missing a relevant paper is usually more costly than retrieving an extra candidate. A person skimming results self-corrects, and a bad first hit costs a glance. An agent uses whatever comes back as its evidence, so a missing paper is invisible to it while an extra one is cheap to ignore.

That also explains why relevance benchmarks built on human judgments are a rough proxy here. They reward results a person would rate topically close, which is a different bar from whether the returned set contains the paper the agent needs to quote.

Two limitations worth carrying forward. Firecrawl ran this benchmark on its own index, so it is a published, reproducible methodology rather than an independent audit. And no other provider here has published an arXivQA number, so the 18% figure describes a comparison against an unnamed provider Firecrawl tested, not a public leaderboard.

Most academic retrieval work reduces to three shapes. I ran the equivalent Firecrawl Research Index operations live while testing these workflows, and the output below is what came back.

The TypeScript examples use the official firecrawl Node SDK and were type-checked against firecrawl@4.32.0. If your agent speaks MCP rather than HTTP, the docs recommend reaching the Firecrawl Research Index through its CLI or MCP server combined with a dedicated research skill.

To give your agent access to the Firecrawl Research Index, you can use the Firecrawl CLI or MCP, combined with Firecrawl's dedicated research skill, which you can install with:

npx skills add firecrawl/skills@firecrawl-research-index

npm install firecrawl@4.32.0

Single-hop retrieval. Find the paper, then pull the passage that answers the question.

import { Firecrawl } from "firecrawl";
 
const firecrawl = new Firecrawl({
  // Falls back to FIRECRAWL_API_KEY. Research works keyless at low volume.
  apiKey: process.env.FIRECRAWL_API_KEY,
  maxRetries: 3,
  backoffFactor: 0.5,
});
 
const found = await firecrawl.research.searchPapers(
  "detecting citation hallucination in retrieval augmented generation",
  { k: 5 },
);
 
for (const paper of found.results) {
  console.log(paper.primaryId, paper.title);
}

Output, top three of five:

arxiv:2601.05866  FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG
arxiv:2605.27700  CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text
arxiv:2603.27752  Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Each result carries both a paperId (the index's internal key) and a primaryId in arxiv:<id>, doi:<id>, or pmid:<id> form. Use primaryId for citations and re-fetching, since it resolves outside Firecrawl.

Passing query switches getPaper from metadata mode into read mode:

const read = await firecrawl.research.getPaper("arxiv:2601.05866", {
  query: "What does FACTUM measure, and what results does it report?",
  k: 3,
});
 
console.log(read.paper.title, "|", read.paper.authors);
for (const passage of read.passages) {
  console.log(`[${passage.score.toFixed(3)}] ${passage.text.slice(0, 200)}...`);
}

One of the returned passages is from the results section:

the full FACTUM framework achieves the highest overall performance, winning 27 of 30 head-to-head comparisons with a peak AUC of 0.737 with the Logistic Regression classifier. Crucially, FACTUM increases 8B model precision from 0.201 to 0.334, which is a 66% relative improvement over the ReDeEP baseline.

The paper's abstract says something related but much less specific: FACTUM reports improvements of up to 37.5% in AUC over the baselines it evaluates.

Both are accurate, and they support different claims. Working from the abstract, an agent can say the method performs better. Working from the passage, it can report 27 of 30 comparisons at 0.737 AUC on NeuCLIR 2024 and point at the sentence that says so.

That is the practical difference full-text retrieval makes. Passage selection matters as much as document ranking, which is the same problem chunking strategies address in a RAG pipeline.

Given a claim, find what supports it and what complicates it, then check the body text before the agent commits to a position.

Take a claim that sounds settled: chain-of-thought prompting only helps sufficiently large models.

const evidence = await firecrawl.research.searchPapers(
  "chain-of-thought prompting is an emergent ability that only helps sufficiently large models",
  { k: 3 },
);

Output:

arxiv:2201.11903  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
arxiv:2212.08410  Teaching Small Language Models to Reason
arxiv:2306.14050  Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step

The first result supports the claim: its abstract describes reasoning abilities that "emerge naturally in sufficiently large language models." An agent that stops at rank one confirms the claim and moves on.

The third result appears to contradict it. Reading the body complicates that initial conclusion rather than reversing it:

const check = await firecrawl.research.getPaper("arxiv:2306.14050", {
  query: "Do small models benefit from chain-of-thought prompting, and at what scale?",
  k: 2,
});

The passage that comes back:

chain-of-thought prompting has only been shown to be beneficial for models of sufficient scale (e.g., with more than 60B parameters Wei et al. (2022b)). In this work, we study whether small language models can be "taught" the capacity for chain-of-thought reasoning by larger language models.

The two papers are describing different things. Taken together, these results suggest the original claim applies to prompting in the setup Wei et al. studied, and should not be generalized to chain-of-thought capability after distillation, where the SCoTD authors report that models between 125M and 1.3B parameters produce useful reasoning chains once trained on a larger teacher's rationales.

Two papers do not settle a literature, and that is not the point here. The point is that the apparent contradiction dissolves once the agent reads past the titles.

An agent working from titles alone would record this as a contradiction. Working from abstracts, it would likely take whichever it saw first. The distinction only appears in the body text. Firecrawl has written about agents that surface this kind of disagreement.

Start from one paper and walk the graph. intent is required, and it does real work: it semantically reranks the structural candidates, so the same seed returns different neighborhoods depending on what you ask for.

const related = await firecrawl.research.similarPapers("arxiv:2601.05866", {
  intent: "detecting and mitigating citation hallucination in RAG",
  mode: "similar", // or "citers" / "references"
  k: 5,
});
 
console.log(`evaluated ${related.poolSize} candidates, truncated=${related.truncated}`);
for (const paper of related.results) {
  console.log(paper.primaryId, paper.title);
}

Output:

evaluated 5 candidates, truncated=false
arxiv:2512.08892  Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders
arxiv:2510.21538  InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generation
arxiv:2504.05324  Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysis
arxiv:2408.15533  LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagation
arxiv:2512.09148  Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment

That gives the agent a coherent cluster on mechanistic hallucination detection from a single seed. Check poolSize and truncated before trusting the shape of a result set: poolSize reports how many candidates were resolved before truncation to k, so a small pool means the neighborhood is thin rather than cut off.

references shows what a paper builds on, and citers surfaces papers published later that cite it. citers identifies subsequent work, but it does not tell you whether those papers reproduce, extend, or contradict the original.

The agent still has to retrieve and read those papers to find out. That is another use for read mode.

The Firecrawl Research Index is scoped to AI/ML, so a production agent needs a path for queries outside that corpus. This function tries the index first and falls back to Firecrawl's general web search for grey literature or out-of-corpus topics.

First the primary path, which returns null rather than throwing whenever it cannot produce grounded evidence:

type Evidence = { url: string; title?: string; text?: string };
 
type Grounded =
  | { source: "research_index"; id: string; title: string; passages: string[] }
  | { source: "web_search"; results: Evidence[] };
 
async function fromResearchIndex(question: string): Promise<Grounded | null> {
  const hits = await firecrawl.research.searchPapers(question, { k: 5 });
  const top = hits.results[0];
  if (!top) return null;
 
  const doc = await firecrawl.research.getPaper(top.primaryId, {
    query: question,
    k: 3,
  });
  // Passages can come back empty when full text is not indexed.
  if (doc.passages.length === 0) return null;
 
  return {
    source: "research_index",
    id: top.primaryId,
    title: doc.paper.title,
    passages: doc.passages.map((p) => p.text),
  };
}

Then the composition, where the fallback lives:

async function findGroundedAnswer(question: string): Promise<Grounded> {
  try {
    const hit = await fromResearchIndex(question);
    if (hit) return hit;
  } catch (err) {
    console.warn("research index unavailable, falling back:", err);
  }
 
  const web = await firecrawl.search(question, {
    limit: 5,
    sources: ["web"],
    scrapeOptions: { formats: ["markdown"] },
  });
 
  return {
    source: "web_search",
    results: (web.web ?? []).flatMap((r) => {
      if (!("markdown" in r) || typeof r.markdown !== "string") return [];
 
      const url = r.metadata?.sourceURL ?? r.metadata?.url;
      if (!url) return [];
 
      return [{ url, title: r.metadata?.title, text: r.markdown }];
    }),
  };
}

Four details worth keeping when you adapt this.

scrapeOptions is the part that makes the fallback useful. Firecrawl's search docs are explicit that without it you get url, title, and description only. Passing formats: ["markdown"] returns page content alongside each result, which gives the agent evidence instead of only a link.

It costs more per call, so if you only need to rank candidates, leave it off.

With scrapeOptions enabled, search results can include scraped Document data with markdown and metadata. The fallback keeps only results that actually contain Markdown and a source URL, because a URL without retrieved evidence does not satisfy the grounded return contract.

The empty-passages check matters more than it looks. getPaper resolves metadata for papers whose full text is not indexed, so passages can come back empty on an otherwise successful call. Without the check, the agent returns a paper with no evidence attached and treats it as a hit.

Both paths return a source field, so the caller always knows which layer produced the answer. Logging that makes retrieval problems much easier to diagnose later.

The catch covers transport failure, not empty results. An empty result set is a valid answer meaning the corpus does not cover this question, and it should fall through to web search rather than raise.

The client above also enables bounded retries for transient Firecrawl failures. If you compose this with arXiv or Semantic Scholar, put those providers behind their own concurrency and rate limiters. A retry policy cannot compensate for deliberately exceeding arXiv's one-request-per-three-seconds rule or a dedicated Semantic Scholar limit.

If your agent needs coverage beyond the paper corpus, the Firecrawl Search endpoint is the layer handling that fallback path.

In practice, academic retrieval usually works better as a combination of sources. These five fail in different places, which makes them reasonable to compose.

A reasonable default for an AI/ML agent is Firecrawl Research Index as primary, OpenAlex when the query is clearly out of domain, and Firecrawl /search for material discussed on the web but not published as a paper.

For a broader research stack, the deep research API comparison covers provider tradeoffs. Web Search and Deep Research for AI Agents explains how the general web layer fits underneath.

Budget the combination before you ship it. With a free OpenAlex key, $1 per day is roughly 1,000 keyword searches, 10,000 list or filter calls, or 100 cached-content downloads if each operation is used on its own. A real agent mixes those operations, so model cost per question rather than assuming every request has the same price.

Reliability here is not about returning more results. It comes down to three properties you can test:

• Reproducible retrieval: the agent preserves the query, source, and stable paper identifiers needed to re-fetch and verify the evidence it used, even if rankings change as the index is updated.
• Verifiable citations: the agent returns a passage it retrieved rather than a sentence it generated about a paper it only saw the title of.
• Sufficient freshness: the index reflects recent work, so the answer is not quietly describing an older state of the literature.

A stronger retrieval layer does not eliminate hallucinations, but it gives the agent better evidence to work from and makes its citations much easier to verify. Some failures that look like reasoning problems are actually retrieval failures, and those are the ones this layer can address.