Compare experiments

Compare experiment runs to decide whether a prompt, model, retrieval, or code change is ready to ship. Start with aggregate scores, then inspect cases that got worse and open their traces to understand the cause.

Experiment runs can use Langfuse datasets or local data. See Evaluate an existing application to create two comparable runs in Python or TypeScript, or explore the example project.

Choose comparable runs

Open Experiments, select the runs to compare, and open the comparison view. Choose the reviewed release run as the baseline.

For a release decision, use the same dataset version and evaluator definitions for baseline and candidate. Record the application commit, prompt or model version, and evaluator version in experiment metadata. A dataset version fixes the test data; it does not make model outputs deterministic.

You can compare runs from different data sources, but first check that the cases represent the same inputs and expected outputs. A missing case is not a passing case. For local data, preserve case identifiers in your test artifacts so your CI policy can match cases explicitly.

Choose the release policy before interpreting the results: must every required case pass, or must the candidate preserve all previously approved passes? Known failures need explicit acceptance; a newer run is not automatically an approved baseline.

Inspect scores and outputs

Review score, cost, and latency differences to identify tradeoffs. An improvement in average quality can hide a regression on a critical case.

Both runs above have 50% accuracy. The average alone does not tell you whether the candidate is safe to release.

Use score thresholds in the comparison view to narrow the results, then inspect the baseline and candidate outputs side by side. Check the score explanation against the output. If an evaluator failed or returned no result, resolve that error before treating the comparison as complete.

Investigate and review failures

Open the trace for a failing item. Inspect the application output and the intermediate retrievals, model calls, or tools that produced it. For example, an incorrect refund window might come from outdated retrieved policy text or from the model ignoring the correct policy.

Separate application failures from evaluator mistakes. A good paraphrase can fail a string check; a fluent answer can still contain an unsupported claim. Use human scores to record the review outcome and failure reason. Use annotation queues when review needs to be shared across a team.

If you change the evaluator after review, re-score both versions with the same updated definition. Keep the original experiment metadata so the decision can be reproduced.

Share results with reviewers

Give the reviewer access to the project through organization invitations and roles. A Viewer can inspect results; a Member can add scores and comments. Sending a link does not grant project access.

Send the experiment links with the baseline and candidate run names, their version identifiers, the cases to review, and the release policy. Ask the reviewer to open Experiments, select those runs, and choose the stated baseline. Include any score filters in the handoff so the reviewer can reproduce your selection.

For a QA team reviewing answers, add the relevant experiment item observations to an annotation queue. Include the source material and reference answer they need to judge the output. Reviewers can record a score, explain the failure, and propose a corrected answer without running your code.

Turn the decision into a CI policy

Approve a baseline explicitly and record its identity with the dataset and evaluator versions. Do not automatically replace it with every successful candidate. A useful policy combines minimum aggregate scores with checks for newly failing critical cases and incomplete results.

See Experiments in CI/CD for an example. The baseline approval and gate policy live in your repository; choosing a baseline in the comparison UI does not configure a CI gate.

---

---