Custom Models

Fine-Tuning & Model Distillation — the right-sized model for the task, not the biggest one.

Most tasks don't need your biggest, most expensive model — they need one trained on your data for exactly what you're asking it to do. We fine-tune and distil models that are faster, cheaper, and often more accurate than the general-purpose model they replace.

See the case study ↓
10×Lower latency*
60%Inference cost cut*
+4ptTask accuracy*
All evals passing
elhaa · model training pipeline
1Task & Data Audit
2Dataset Preparation
3Train & Evaluate vs Baseline
4Deploy & Version
Benchmarked vs baseline
What's included

A complete training pipeline, not a one-off model.

Data preparation, evaluation, and versioning — the full system that makes a custom model maintainable.

Training dataset pipeline

Clean, de-duplicated, and structured examples with clear evaluation splits — versioned like code.

02

Fine-tuning & distillation

Domain adaptation of open or proprietary models, or distillation into a smaller, faster model.

03

Head-to-head evaluation harness

Every candidate model benchmarked against your current baseline before it ships.

04

Regression monitoring

Ongoing checks that the new model doesn't quietly get worse on cases it used to get right.

05

Deployment & rollback

Shipped behind your existing interface, with a clean path back to the baseline if needed.

Engineering deep dive

How it's actually trained.

The training topology and the evaluation pattern — not a slide about “AI customisation.”

1Labelled Training & Evaluation Dataset
2Fine-Tuning / Distillation Run
3Head-to-Head Evaluation vs Baseline Model
4Versioned Deployment with Rollback
finetune-eval.ts
// elhaa Fine-Tune Evaluation Harness
const comparison = await elhaaEval.compare({
  baseline: 'gpt-4o',
  candidate: 'ft:gpt-4o-mini:elhaa:v3',
  evalSet: 'domain-tasks-v2',
  metrics: ['accuracy', 'latencyMs', 'costPer1k']
});
Case study

Support Ticket Triage, Distilled to a Fraction of the Cost

B2B SaaS · Series B software company

The challenge

A general-purpose model classified and routed support tickets accurately, but at a per-ticket cost that made the feature a net loss at the company's actual support volume.

The approach

We fine-tuned a small open-weight model on 4,000 historically-labelled tickets, distilled the classification behaviour from the original large model, and shipped it behind the same routing interface with a rollback path.

Historical ticket export → label cleaning → distillation from large model → fine-tune → head-to-head eval → deploy
91%Lower inference cost
+2.1ptAccuracy vs original model
<80msP95 latency

*Illustrative example based on a representative engagement.

The difference

The typical approach vs the elhaa approach.

Typical approach
With elhaa
Model choice
One large general-purpose model for everything
Small, fine-tuned model for the task that needs it
Latency & cost
Pay large-model prices for simple tasks
10x lower latency and cost on distilled models
Data exposure
Prompts sent to a third-party API every time
Fine-tuned weights can run inside your boundary
Consistency
Prompt engineering drifts across releases
Behaviour baked into weights, versioned like code
How the engagement runs

Four steps from audit to a versioned model.

1

Task & data audit

Identify which task justifies a custom model, and whether you have enough labelled data.

2

Dataset preparation

Clean, de-duplicate, and structure training examples with clear evaluation splits.

3

Train & evaluate

Fine-tune or distil candidate models, benchmarked against the current general-purpose baseline.

4

Deploy & version

Ship the winning model behind the same interface, with rollback to the baseline if needed.

How success is measured

Agreed in week one, on a dashboard by go-live.

Quality

Task accuracy vs baseline

Fine-tuned model measured head-to-head against the general-purpose model it replaces.

Efficiency

Latency & cost per call

The core promise of distillation — tracked explicitly, not assumed.

Safety

Regression rate

Share of previously-correct cases the new model gets wrong — must trend to zero.

Ownership

Retrain cadence

How often the model is refreshed as your data and task drift.

Works with your tools

Typical systems & standards.

OpenAI fine-tuningAnthropic & open-weight modelsLoRA / QLoRAWeights & BiasesHugging FaceVector eval harnesses
Who's involved

Small teams on both sides.

From elhaa
  • ML engineerOwns the training pipeline and evaluation harness.
  • Data curatorBuilds and cleans the training and evaluation datasets.
  • MLOps engineerHandles deployment, versioning, and rollback.
From your side
  • Domain expertLabels or reviews training examples for correctness.
  • Data ownerApproves what training data can be used and where.
  • Engineering contactWires the new model behind your existing interface.
FAQ

Questions about Fine-Tuning & Model Distillation.

It varies by task, but useful results often start from a few hundred well-labelled examples; a few thousand gives more headroom. The task and data audit tells you exactly where you stand before we commit to training.

It can be — a fine-tuned open-weight model can run entirely inside your own infrastructure, so no prompts or data ever leave your boundary. We'll tell you honestly if your task doesn't justify that complexity.

Fine-tuning adapts a model to your specific task and data. Distillation trains a smaller, cheaper model to mimic a larger one's behaviour on that task. We often do both together: distil for cost, fine-tune for accuracy.

It's benchmarked head-to-head against your current baseline on a held-out evaluation set before it ever reaches production — no deploy without a measured win.

We set a retrain cadence appropriate to how fast your task drifts, and monitor for regression so you know when a refresh is actually needed rather than guessing.

Sounds like your situation?

A 30-minute call. We'll tell you honestly whether this is the right solution — and what it would take.

Explore other services