Catalogue
/
Artificial Intelligence and Applied AI
/
Langfuse for Tracing, Evaluation, and Monitoring of Production AI Systems

Langfuse for Tracing, Evaluation, and Monitoring of Production AI Systems

A practical two day technical course for participants who are new to Langfuse and want to build observability, evaluation, prompt management, and monitoring into production AI applications. The agenda progresses from tracing and instrumentation through prompt management, datasets, evaluations, experiments, production monitoring, and operational integration. The course focuses on practical patterns for understanding AI application behavior, measuring quality, detecting regressions, and improving systems systematically.

What will you learn?

You will learn how to instrument AI applications with Langfuse, analyze traces and generations, manage prompts centrally, and evaluate application quality using datasets, scores, and experiments. You will work with online and offline evaluation concepts and learn how production observations can be turned into repeatable test cases. You will also understand how to monitor quality, latency, token usage, and cost while integrating Langfuse into an existing AI engineering workflow.

  • Instrument AI applications and analyze LLM traces, observations, sessions, and generations
  • Manage and version prompts while connecting prompt changes with production behavior
  • Build evaluation datasets and compare models, prompts, and application changes through experiments
  • Monitor production AI systems using quality, performance, usage, and cost signals

Requirements:

  • Technical background in software development, AI engineering, data engineering, or a related field
  • Basic programming knowledge, preferably Python or JavaScript
  • General familiarity with LLM APIs and modern AI application concepts
  • Previous Langfuse experience is not required

Course Outline*:

*We customize the course outline and content to your specific needs and relevant use cases.

Module 1: Langfuse architecture and AI observability fundamentals

  • Understanding the role of Langfuse in the AI application lifecycle
  • Navigating projects, traces, observations, generations, sessions, and users
  • Understanding how requests flow from an AI application into Langfuse
  • Distinguishing application observability, evaluation, monitoring, and prompt management

Module 2: Instrumenting LLM applications

  • Adding tracing with Langfuse SDKs and supported integrations
  • Capturing model calls, inputs, outputs, metadata, usage, latency, and cost
  • Structuring nested observations for chains, retrieval, tools, and agent workflows
  • Using tags, metadata, users, and sessions to make traces easier to analyze

Module 3: Analyzing traces and application behavior

  • Following end-to-end execution across model calls, retrieval steps, and tools
  • Identifying slow, expensive, failed, or low-quality interactions
  • Comparing behavior across models, users, releases, and application paths
  • Turning useful production traces into inputs for further evaluation and testing

Module 4: Prompt management and versioning

  • Creating and managing prompts centrally in Langfuse
  • Working with prompt versions, variables, labels, and deployment stages
  • Connecting prompt versions to traces and generations
  • Comparing prompt behavior and preparing controlled prompt changes for evaluation

Module 5: Evaluation concepts and scoring

  • Understanding scores as a common representation for AI quality measurements
  • Applying human feedback, programmatic checks, user feedback, and LLM-as-a-Judge evaluation
  • Defining useful quality dimensions such as correctness, relevance, helpfulness, and safety
  • Attaching evaluation results to traces, observations, sessions, and experiment runs

Module 6: Datasets and experiments

  • Building evaluation datasets from curated examples and production traces
  • Defining inputs, expected outputs, metadata, and reusable test cases
  • Running experiments to compare prompt, model, retrieval, or application changes
  • Comparing experiment results and identifying regressions before deployment

Module 7: Production monitoring and continuous evaluation

  • Monitoring quality scores, latency, usage, token consumption, and cost over time
  • Applying online evaluation to selected production traffic
  • Identifying recurring failure patterns and segments with degraded performance
  • Feeding production findings back into datasets and offline evaluation workflows

Module 8: Production integration and operational readiness

  • Integrating Langfuse with existing AI frameworks, services, and observability environments
  • Understanding OpenTelemetry integration and broader monitoring architectures
  • Comparing Langfuse Cloud with self-hosted deployment options
  • Building a practical operational workflow covering tracing, prompt changes, evaluation, release decisions, monitoring, and continuous improvement

Hands-on learning with expert instructors at your location for organizations.

4,122€*
Graph Icon - Education X Webflow Template
Level:
advanced
Clock Icon - Education X Webflow Template
Duration:
14
Hours (days:
2
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Immersive hands-on experience in a dedicated setting
*Price can range depending on number of participants, change of outline, location etc.

Master new skills guided by experienced instructors from anywhere.

3,097€*
Graph Icon - Education X Webflow Template
Level:
advanced
Clock Icon - Education X Webflow Template
Duration:
14
Hours (days:
2
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Reduced training costs
*Price can range depending on number of participants, change of outline, location etc.

Upcoming Sessions

28-29 Jan 2027
Madrid
10-11 Feb 2027
Zurich
25-26 Feb 2027
Barcelona
8-9 Apr 2027
Zurich
15-16 Apr 2027
Madrid
6-7 May 2027
Berlin
1-2 Jun 2027
London
1-2 Jul 2027
Munich

Can't find a suitable date? Get in touch and we'll arrange one that works for you.

We use cookies to improve site navigation, analyse how the site is used, and support our marketing. You can accept all cookies, reject non-essential ones, or choose individual categories. Read our Cookie Notice