Evaluating and improving agents with Langfuse

Hey Emre Elbeyoglu,
 
Using evaluators to find interesting traces shows you where to look. Building out a suite of evals
[https://langfuse.com/academy/evaluate?ref=onboarding_emails_v2_5_datasets_experiments]to reliably test your agent system allows
you to continuously improve and check for regressions before you ship, and through that, close the loop
[https://langfuse.com/academy/datasets?ref=onboarding_emails_v2_5_datasets_experiments].
 



 
Create your first dataset
[https://langfuse.com/docs/evaluation/experiments/datasets?ref=onboarding_emails_v2_5_datasets_experiments]
A dataset is a set of test cases you want to repeatedly evaluate your agent against. The first dataset can be small. 15-30
examples that represent what your application handles with clear expectations on the output.
 
Run your first experiment
[https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk?ref=onboarding_emails_v2_5_datasets_experiments]
Experiments allow you to test variants of your system. A different prompt, a cheaper model, another harness. Each run gets
evaluated and you can compare the different runs of your system.
 
Run experiments in your CI/CD
[https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd?ref=onboarding_emails_v2_5_datasets_experiments]
Some happy paths of your application need to always work. Check those as part of your CI/CD by including an experiment run as a
deployment gate.
 
If anything is not covered here, just hit reply, we read every message. Or book a call
[https://cal.com/team/langfuse/welcome-to-langfuse] and we'll look at your setup together.
 
Best,
Marc
 
Co-founder, Langfuse (part of Clickhouse)
Links: Roadmap [https://langfuse.com/docs/roadmap?ref=onboarding_emails_v2_5_datasets_experiments], Docs
[https://langfuse.com/docs?ref=onboarding_emails_v2_5_datasets_experiments], Talk to us
[https://langfuse.com/talk-to-us?ref=onboarding_emails_v2_5_datasets_experiments], Example Project
[https://langfuse.com/docs/demo?ref=onboarding_emails_v2_5_datasets_experiments], Langfuse Academy
[https://langfuse.com/academy?ref=onboarding_emails_v2_5_datasets_experiments]
 
You are receiving this email because you opted in to receive email from the list Onboarding Journey from Langfuse
Langfuse, Langfuse GmbH, Oranienburger Strasse 91, 10178, Berlin, Germany
Unsubscribe [https://app.loops.so/unsubscribe/cmvbfvd2a3ml801xp0kithdxg/3a766fb415be2ba2902872a3068ec838ac231bc9d61e1a34fa826173ae0416fd095587c45c24acb1592f4ffe2104a055aeb6c9ba4ddade8c7a60b78f]

Accession

nse/langfuse/2026-10-16/4298

From

Marc Klingen · marc@updates.langfuse.com

Sent

16 October 2026 · 20:00 UTC