How to Fine-Tune Laya on a Mac with Data from Firecrawl Alexandria

TL;DR:

• Laya is an open source, 421M-parameter decision model (Apache 2.0) that runs on a laptop. Stock, it scored 44.0% on a developer search relevance task where Jev scored 74.3%.
• Training data came from Firecrawl's Developer Index through Alexandria: 298 developer queries, 10 results each, 2,980 query and result pairs for about 600 credits.
• Claude Opus 5.5 labelled every pair through the Batches API in 16 minutes for about $4.25. 60 queries (600 pairs) were held back as a test set.
• Fine-tuning ran in Python on an M5 MacBook Pro: 98.5 minutes, 21.4 GB peak memory, $0 in API costs. The trained model then runs from Python or TypeScript.
• Fine-tuned Laya reached 68.7% relevance accuracy and 76.2% on spotting a fix, against Jev's 74.3% and 77.2%. It answers in 53 ms locally versus Jev's 263 ms.
• Gotcha: laya 0.4.0's trainer crashes on Mac GPUs with Passed CPU tensor to MPS op. A three-line patch fixes it.
• Calibration still left the model over-confident, so don't gate on its confidence scores without checking them first.

To fine-tune Laya, you give its trainer a file of labelled examples and let it retrain the model for your one task. Laya is small enough that this runs on a MacBook. The hard part is getting thousands of labelled examples, so this walkthrough uses Firecrawl Alexandria for the data and Claude for the labels.

The example task is a search filter for coding agents: does this search result answer the developer's question? Stock Laya got 44% right. One fine-tune on a MacBook later it scored 69%, against 74% for TypeSafe's Jev.

Laya is an open source decision model by Convai Innovations. A decision model doesn't write text. You give it some text and a question with fixed answers, and it returns a probability for each answer. Jev, by TypeSafe, does the same job from the cloud, while Laya's 421M parameters fit on a laptop.

Laya was released after Jev, but the idea behind it is older. Its creator, Nandakishor M, published a paper on a similar fast, probability-based decision model in March 2025, about 18 months before Jev.

Laya is published on GitHub and Hugging Face by Convai Innovations. A parameter is one of the internal numbers a model adjusts during training. Laya's English checkpoint is built on ModernBERT-large and downloads as about 840 MB. Jev's size isn't public, but the strongest open Jev-style models on the Jev Decision Index have around 27 billion parameters.

Laya also uses the same request shape as Jev: a state string plus a questions object. The same questions work in both, from TypeScript too:

import { TypeSafeClient } from "@typesafe-ai/sdk";
import { Agent } from "laya-ts";
 
const query = "npm ERESOLVE unable to resolve dependency tree peer dependency";
const title = "Unable to resolve dependency tree error in Angular while installing npm packages";
const url = "https://www.angularjswiki.com/angular/unable-to-resolve-dependency-tree-error-in-angular-while-installing-npm-packages/";
const passage = "It's a dependency conflict error, so run the command with --force or --legacy-peer-deps flag.";
 
const state = `Query: ${query}\n\nResult: ${title}\nURL: ${url}\n\n${passage}`;
const questions = {
  relevance: {
    type: "score",
    instructions: "How well does this search result answer the developer's query?",
    criteria: [
      "Unrelated to the query",
      "Related topic, but does not answer the query",
      "Directly answers the query",
    ],
  },
};
 
const jev = await new TypeSafeClient().systemOne({ state, questions });
const laya = await (await Agent.load("./models/finetuned")).predict(state, questions);

Both calls return an answers object with a probability for each level. Jev's call goes to TypeSafe's API. Laya's runs locally through laya-ts, Laya's TypeScript package, which isn't on npm: you build it from the Laya repo. It also needs the model in ONNX format, so export the fine-tuned model once with the repo's script:

python laya-ts/scripts/export_onnx.py --model-dir ckpt/laya-devidx --out-dir models/finetuned

ckpt/laya-devidx is where training saves the model, and models/finetuned is the folder the example loads. For code that calls Jev over HTTP, Laya's README says laya-serve exposes the same POST /v1/systemone endpoint on port 8000. That server wasn't tested in this run.

For more on how Jev itself works, see what Jev is and how TypeSafe's decision model works.

Every number in this article comes from one test. 298 developer questions were searched on Firecrawl's Developer Index, 10 results each. Claude graded all 2,980 results, Laya trained on 2,380 of them, and the other 600 became the test set. Those 600 come from 60 questions Laya never saw during training.

For each result, the model answers two questions. Relevance asks whether the result is unrelated, related but doesn't answer, or directly answers the question. has_fix asks whether it contains a working fix. Accuracy means agreeing with Claude's grade.

Note: Claude's grades haven't been checked by hand, so "accuracy" means agreement with Claude, not with a human.

Firecrawl Alexandria is a catalogue of data tools that agents and scripts can discover and call through Firecrawl, drawing on more than 113 million indexed sources. The Developer Index is Firecrawl's own index of GitHub issues, merged pull requests, READMEs, and documentation pages, built for coding questions. Each result comes with its URL, title, and matched passages.

Alexandria works from the Firecrawl CLI, the MCP server, the official ChatGPT plugin and Claude plugin, and the language SDKs. The Alexandria launch covers the catalogue, and the Developer Index launch post covers what the index holds.

Note: the Developer Index works without an API key up to a daily keyless cap. A few thousand searches will outgrow it, and a free account adds 1,000 credits at 2 credits per 10 results.

Fine-tuning means taking a model that is already trained and training it a little more on examples from one task. The model keeps what it already knows, such as how to read English and code, and learns your standards for one job. Think of a smart new hire who gets a stack of graded examples on day one: they aren't learning to read, just learning what you count as a good answer.

Laya needs it because stock Laya is weak on narrow tasks. On the 600 test results, it was only 13 points better than guessing at random:

Stock Laya can't tell "on topic" from "answers the question". Every result was graded on three levels: unrelated, related but doesn't answer, and directly answers. Stock Laya got only 20% of the "related but doesn't answer" results right, which made up half the test set.

A "related but doesn't answer" result is about the right library or error but doesn't solve the problem. A typical one is a GitHub issue where someone reports the exact same error and nobody posts a fix. A keyword match ranks it highly, but a developer who opens it learns nothing new. A filter that lets these through wastes a scrape and fills an agent's context with noise.

Jev handles the middle level three times better than stock Laya. The "unrelated" row has only 10 examples, so its percentages are mostly noise.

You need Python with the laya package, labelled data in a JSONL file, and a GPU for full fine-tuning. Laya's trainer runs on an Apple Silicon Mac, an NVIDIA GPU, or Kaggle's free T4 GPUs through Laya's notebook. This test used an M5 MacBook Pro.

Training is Python only. Laya's TypeScript package runs a trained model but can't train one.

python -m pip install laya

Training runs for 4 epochs by default. An epoch is one full pass through the training data, so Laya studies every example 4 times.

Note: keep a laptop plugged in. Training holds the GPU at full load, and a 61 W charger couldn't keep the battery above 10% during this test.

Collecting the data takes one Developer Index search per query. Claude wrote 298 developer queries covering real errors and how-tos in JavaScript, TypeScript, Python, Rust, Go, and infrastructure. Each query ran once and returned 10 results, which gave 2,980 query and result pairs.

A single search from the Firecrawl CLI looks like this:

firecrawl scrape --alexandria firecrawl-developer-index/search \
  --options '{"query":"fix Next.js hydration error caused by Date.now()","k":10}' \
  --json

The response holds a results array. Each result has an id such as issue:owner/repo#123, a url, a title, and passages with the matched text in markdown.

The collection script runs that command for every query and writes one row per result:

# collect.py
 
def search(query):
    opts = json.dumps({"query": query, "k": 10})
    p = subprocess.run(
        ["firecrawl", "scrape", "--alexandria", "firecrawl-developer-index/search", "--options", opts, "--json"],
        capture_output=True, text=True, timeout=120,
    )
    a = json.loads(p.stdout[p.stdout.index("{"):])["data"]["alexandria"][0]
    return a["data"]["results"]

search() returns the 10 results for one query. The full script adds retries and writes each result to results.jsonl with its query, rank, URL, title, and passages. The CLI call works the same from any language, so a TypeScript version only needs to swap subprocess.run for a child process call.

58 of the 2,980 results came back with no matched passage, only a title and URL.

Note: the account used here allowed about 12 requests per minute, and rejected calls counted too. Three workers at one call every 18 seconds each stayed under it, so 298 searches took about 30 minutes.

Labelling means giving every training example its correct answer. A large model like Claude Opus can grade thousands of examples quickly, and the small model then learns from those grades. This example used Claude Opus 5.5. The pattern is called distillation: the big model teaches once, then the small model does the job on its own, locally and for free.

Every pair got two answers. Relevance is 0 (unrelated), 1 (related but doesn't answer), or 2 (directly answers). has_fix is true when the shown text contains a concrete fix, solution, or how-to. Claude also wrote a one-line reason for each grade.

# label.py
 
SCHEMA = {
    "type": "object",
    "properties": {
        "relevance": {"type": "integer", "enum": [0, 1, 2]},
        "has_fix": {"type": "boolean"},
        "reason": {"type": "string"},
    },
    "required": ["relevance", "has_fix", "reason"],
    "additionalProperties": False,
}

SCHEMA is passed as structured output on every request, so each grade comes back as valid JSON with exactly those three fields. The script submits all 2,980 pairs as one job through Anthropic's Message Batches API, with the grading instructions cached in the system prompt.

Almost half the results were on topic but didn't answer the query. That is the hard case a relevance filter has to catch.

Use JSONL, one example per line, with three fields: state (the text the model reads), questions (what to ask), and expected (the correct answers). laya-train also accepts CSV, but it turns a CSV into a single multiple-choice question per row. Two questions per result, including a ranked score and a yes/no, need JSONL.

A score question picks one level from an ordered list. A noul is a yes/no question answered with a probability, a name TypeSafe introduced for Jev. One example from the training set, formatted for reading, looks like this:

{
  "state": "Query: fix Next.js hydration error caused by Date.now() in a server component\n\nResult: Next.js hydration error in production | Afterbuild Labs\nURL: https://afterbuildlabs.com/fix/nextjs-hydration-error-production\n\nThe fix is to render a stable value on the server and update it inside useEffect after mount.",
  "questions": {
    "relevance": {
      "type": "score",
      "instructions": "How well does this search result answer the developer's query?",
      "criteria": [
        "Unrelated to the query",
        "Related topic, but does not answer the query",
        "Directly answers the query"
      ]
    },
    "has_fix": {
      "type": "noul",
      "instructions": "The result contains a working fix, solution or concrete how-to for the query"
    }
  },
  "expected": { "relevance": 2, "has_fix": true }
}

The state packs the query, the result's title and URL, and its matched passages into one string. The median state was 124 tokens, and Laya's English checkpoint reads up to 512.

The test set was split off by query, never by result: 238 queries for training and 60 for testing. If one query appeared in both, the test would measure memory instead of skill.

Laya's trainer runs as a single command, pointed at your JSONL file and a base model. It trains for 4 epochs, calibrates the model's confidence, and saves the fine-tuned model. On laya 0.4.0 it crashes on Mac GPUs, so this test called the trainer from Python with a small patch.

laya-train --data train_laya.jsonl --base english --out ./laya-devidx

The Mac crash is Passed CPU tensor to MPS op. The fix is to move the model to the GPU before Laya scores it:

# finetune.py
 
import laya.train
from laya.train import TrainConfig, finetune
 
_calibration_records = laya.train.calibration_records
 
 
def _calibration_records_on_device(model, tok, items, device, *args, **kwargs):
    model.to(device)
    return _calibration_records(model, tok, items, device, *args, **kwargs)
 
 
laya.train.calibration_records = _calibration_records_on_device
 
finetune(data="data/train_laya.jsonl", model_dir="english", output_dir="ckpt/laya-devidx",
         config=TrainConfig(epochs=4, micro_batch=8, grad_accum=8, calib_max=400, seed=42,
                            gradient_checkpointing=True, amp=False),
         device="mps")

On the M5 MacBook Pro, the run took 98.5 minutes, about 24 minutes per epoch, and peaked at 21.4 GB of memory. The fine-tuned model is saved to ckpt/laya-devidx.

An epoch is one full pass through the 4,360 training items, and Laya was tested on the 600 held-out results after each of its 4 epochs. The first epoch did most of the work, lifting relevance accuracy from 44.0% to 64.2% on the 600 test results. Each later epoch added 1 to 2 points, ending at 68.7%. Accuracy was still rising after epoch 4, so more data would likely help more than more epochs.

F1 combines precision and recall for the "directly answers" decision into one score from 0 to 1, where higher is better. It's explained in the comparison with Jev below.

Temperature calibration adjusts a model's confidence scores so they match how often it's actually right. A model that says "90% sure" but is right 75% of the time is over-confident, and a higher temperature softens its scores. After training, Laya fits one temperature per question type on 400 training items it holds back for this.

Calibration only changes how confident the model sounds. It never changes which answer it picks.

In this test it didn't fully work. Both fitted temperatures hit Laya's cap of 5.0, the most softening it allows, and the model was still more confident than accurate:

Treat this fine-tuned model's confidence scores as rankings, not probabilities, and don't gate on Laya's min_confidence without checking it on your own data.

Jev still wins on relevance: 74.3% against fine-tuned Laya's 68.7% on the same 600 held-out results. On spotting a fix the two are effectively tied, 77.2% and 76.2%. Fine-tuned Laya answers in a median 53 ms on a local GPU, about 5 times faster than Jev's 263 ms over the network.

Precision is the share of results a model called "directly answers" that really did. Recall is the share of real answers it found. Jev's lead is mostly recall: it catches 90% of results that answer the query, Laya 77%, at about the same precision.

Fine-tuning fixed most of stock Laya's weak spot, the "related but doesn't answer" level, where it now edges out Jev:

The "directly answers" call is the one an agent acts on: scrape the result or skip it. Run as a filter on the 600 test results, it looks like this:

Both gaps in accuracy are real, not noise. The 10 results from one question aren't independent of each other, so the test below treats each of the 60 questions as one unit:

Individual results still go either way. On the query "python multiprocessing pickling error with lambda", the result was a Python bug report showing the same error with no fix:

Note: that example was picked because Jev gets it wrong. Overall, Jev still wins more of the disagreements.

For a wider look at how decision models compare, see OpenAI's Decisions API vs Jev.

Fine-tuned Laya takes a median 53 ms per decision on a Mac GPU, and Jev takes 263 ms including the network round trip. On CPU through TypeScript and ONNX, Laya takes 140 to 185 ms. The first Laya call is slow: a one-time ~840 MB download, then 3.8 seconds to load.

The Python and Jev times are the median time per call across the 600 test results, measured by the benchmark script on the M5 MacBook Pro. One call answers both questions. Jev's time includes the round trip to TypeSafe's API. The TypeScript range comes from a handful of calls in the side-by-side demo, which runs Laya on the CPU through ONNX.

The cold start times were measured once, by hand, on the same Mac. Load the model once and keep it running. Jev ran all 2,980 pairs in 135 seconds with 6 calls in parallel.

Use Jev when you need good answers across many different tasks without training anything. Jev has more general knowledge, and it beat fine-tuned Laya even on the task Laya was trained for. Fine-tune Laya when the task is narrow and you need local, private, or offline decisions with no rate limits.

Don't pick Laya to save money:

Narrow jobs suit a fine-tuned Laya model: filtering job listings, finding research papers, sorting product reviews, or picking which results an agent scrapes. Each needs a dataset, and Firecrawl Alexandria or Firecrawl Search is where to start collecting one. For fine-tuning generative models instead, see building a custom instruction dataset and fine-tuning Gemma 3 with Firecrawl and Unsloth.

Laya's own limits also shape which tasks fit. Its README says that past about 20 options per question, each option gets trimmed to fit the model's token budget. Accuracy on long documents also varies beyond about 4,000 tokens. The Laya fine-tuning guide covers the trainer's other options.

Fine-tuning Laya turns a few thousand labelled results into a fast local decision model. Firecrawl's Developer Index and a big model's grades build that dataset in an afternoon.

Check out the Firecrawl docs to go deeper.