I changed a prompt in an LLM scoring system and the scores shifted. I had no way to prove whether that made things better or worse. Getting out of that blind spot took me through tool comparisons and community debates.

Why I went looking for eval tools

The system uses a language model to grade exam responses and write feedback.

It works. The model reads inputs, generates commentary, returns scores. Looks fine.

Then I tweaked a prompt.

After the change, scores shifted. The tone of the feedback felt different. But "felt" is a dangerous word in production. I couldn't tell whether results had improved or degraded, along which dimensions, or how many cases were affected.

The uncomfortable part: the system was live, and I was shipping changes blind.

This is an engineering problem. I wrote the prompt; how do I know it's any good?

The seven dimensions of an LLM product

Mapping the problem took a while, but the territory comes down to seven dimensions every LLM product has to care about:

  1. Model selection — Which LLM to use
  2. Prompt engineering — How to write, manage, and iterate prompts
  3. Output quality — Are the results correct and good?
  4. Cost — How much API spend is this burning?
  5. Latency — How long are users waiting?
  6. Reliability — Is the system dependable?
  7. Safety — Hallucinations, bias, data leakage

Each dimension has an MVP tier and a serious tier. For quality, the MVP is 10 golden examples you manually check. The serious tier is LLM-as-Judge, automated eval pipelines, and CI gates.

My take: start with the minimum viable quality + safety setup, expand from there. Most teams ship with nothing — being able to see your problems puts you ahead.

For me, dimension three mattered most. When you're building a scoring system, quality is the product.

Langfuse, Braintrust, and Datadog compared

Once I decided to build an eval practice, the next question was which tools. That turned out to be more painful than expected.

Langfuse: the default choice, mediocre evals

Langfuse is the default answer in LLM observability right now. 33k+ GitHub stars. Acquired by ClickHouse in early 2026. Khan Academy and Canva use it.

The tracing is genuinely good. Every LLM call — input, output, token count, latency, cost — captured and filterable by user, session, prompt version. Prompt management works too: version your prompts in the UI, the SDK caches them, and you can change prompts without redeploying code.

The eval experience is mediocre. You write code to run experiments. Other platforms let you click and compare on one page; here you can't.

Braintrust: better evals, weaker production monitoring

Braintrust just raised an $80M Series B and positions itself as evaluation-first.

The eval UX is better: one-page experiment comparison, a Playground, and a generous free tier — 1M spans/month, unlimited users. Production monitoring is a step below Langfuse. It's more of a lab than a factory.

Datadog: strong generalist, blind to LLMs

I considered Datadog. Great general-purpose tracing. It doesn't understand LLMs — no token cost calculation, no prompt management, no eval. The difference between a general practitioner and a specialist.

What the community says

Reddit discussions made the tradeoffs concrete. On r/LangChain, Langfuse was called the "most serious self hosted open source platform." Someone on r/AI_Agents put the core tension in one line:

"Observability tells you what happened, not how to fix it."

That's the sentence that stuck with me. Another engineer shared a practical approach — instead of dashboard alerts, they pin 8 canary scenarios and rerun them after every prompt change:

"Caught more drift this way than langfuse alerts ever did."

Data export is a real pain point. As one redditor put it:

"Getting the data OUT to do cross-session analysis is a pain."

Hard to argue with that one.

Side-by-Side Comparison

DimensionLangfuseBraintrustDatadog
Tracing★★★★★★★★★★★★★
Eval★★★★★★★★★
Prompt Management★★★★★★★★✗
Open Source✓ (MIT)✗✗
Self-hosted✓✗✗
PricingFree (self-hosted)Generous free tierExpensive
LLM-specific✓✓✗

Why I picked Langfuse

I went with Langfuse Self-hosted.

The reasoning:

  1. One platform covers 80% of my needs — Tracing, Prompt Management, and Eval are all there.
  2. MIT license, self-hosted for free — the data stays inside the VPC.
  3. Good data portability — CSV/JSON/API export, official migration cookbook from Cloud to Self-hosted.
  4. Three platforms is too many. One that's good enough beats three that are perfect individually.

Braintrust has a better eval experience, but adding an entire platform for one capability isn't worth the overhead. Datadog knows infrastructure, not LLMs — wrong tool for this stage.

Four opinions I formed

1. Observability tells you what happened, not how to fix it

To borrow the Reddit quote again. Tracing is infrastructure. Most teams plug it in, stare at the dashboard, feel informed, and do nothing. Seeing what happened is the easy part; understanding it is the work.

2. Most teams ship with nothing

I mean that literally. Many AI products run without eval, without tracing, without golden sets. No judgment if your use case tolerates it. But if you're building something where quality matters — a scoring system, a medical assistant, a legal summarizer — you can't operate blind.

3. One platform is enough. Don't collect three.

Langfuse for tracing + prompt management, Braintrust for eval, Datadog for infrastructure — sounds elegant, until you maintain it. My advice: pick the one that covers the most ground and accept its weaknesses.

4. The space is still early

LLM eval moves fast. The tools of 2025 may be irrelevant by 2027. Langfuse itself was just acquired by ClickHouse — the roadmap two years out is anyone's guess. Don't over-invest in any single platform. Keep your data portable.

Building an eval pipeline from scratch

If you're building an LLM product and want to go from "it runs" to "it's reliable," here's the simplest path:

The bare minimum

  • 10 golden examples — hand-labeled correct answers, covering typical and edge cases.
  • Manual check — after changing a prompt, run the golden set and eyeball the results.
  • That's it. No platform, no code, done in an afternoon.

The three stages

Stage 1: Tracing (1–2 days) Wire up Langfuse. Log every LLM call's input/output/cost/latency. For the first time, you'll see what your system is actually doing.

Stage 2: Prompt Management (2–3 days) Move prompts out of your codebase and into Langfuse. Change a prompt without redeploying. Takes effect in seconds.

Stage 3: Eval Dataset (3–5 days) Build your golden set into Langfuse, configure an LLM-as-Judge. Automatically benchmark before every prompt change. Prevent regressions.

Three stages, roughly a week. After that, it's continuous iteration: Observability → Evaluation → Prompt Management → repeat.

What I learned

Diagram: a prompt change from v1 to v2 is run against a golden set, compared on score, tone, cost and latency, and ends in a ship, review or rollback decision.

Eval is ongoing engineering discipline. I'm still somewhere in Stage 1, but at least now I know which direction to walk.

If you're on a similar path, I hope this saves you some time. No profound insights here. This is the thinking behind a real problem.

FAQ

What is LLM Eval and why does my AI product need it?

LLM Eval is the systematic practice of measuring the quality of large language model outputs. Without it, you can't tell whether a prompt or model change improved or degraded results — which is unacceptable for quality-sensitive applications like scoring, medical, or legal systems.

Langfuse vs Braintrust: which is better for LLM evaluation?

Langfuse covers more ground (tracing + prompt management + eval) and is self-hostable; Braintrust has a superior eval UX but lacks production monitoring and self-hosting. For most teams, the broader platform is the pragmatic choice.

What's the minimum I need to start evaluating LLM outputs?

10 hand-labeled golden examples + manual review after every prompt change. No platform required, done in an afternoon. This is the real MVP — automate only after this habit is established.

What is a Golden Set and how do I build one?

A Golden Set is a curated collection of input-output pairs with human-annotated expected results, covering typical and edge cases. Start with 10 representative examples from production data, manually label the expected outputs, and use them to validate changes.

What is LLM-as-Judge?

LLM-as-Judge is a technique where one LLM evaluates the outputs of another. You define scoring dimensions (accuracy, relevance, tone) and the judge model scores outputs automatically — replacing manual spot-checks with scalable, consistent evaluation.

This post is based on my own research process. Here are the key sources:

References