Catalogue
/
Artificial Intelligence and Applied AI
/
Arize Phoenix for LLM Tracing, Evaluation, and AI Observability

Arize Phoenix for LLM Tracing, Evaluation, and AI Observability

A practical two day technical course for participants who are new to Arize Phoenix and want to trace, evaluate, analyse, and debug LLM applications. The agenda progresses from Phoenix architecture and AI tracing through RAG and agent observability, evaluations, datasets, experiments, and systematic failure analysis. The course focuses on using observability and evaluation together to identify problems, test improvements, and make evidence-based changes to AI applications.

What will you learn?

You will learn how Phoenix captures and visualises traces from LLM applications using OpenTelemetry and OpenInference. You will analyse prompts, retrievals, model calls, tool use, latency, and application behaviour, then use evaluations and experiments to measure whether proposed changes improve quality. You will also learn practical debugging workflows for RAG systems and AI agents.

  • Instrument and trace LLM, RAG, and agent applications with Phoenix
  • Analyse traces to identify quality, latency, retrieval, and tool-calling problems
  • Create and apply automated and human evaluation workflows
  • Use datasets and experiments to compare application changes and investigate regressions

Requirements:

  • Technical background in software development, AI engineering, data science, or a related field
  • Comfortable working with Python, TypeScript, or Java for application instrumentation
  • Familiarity with data structures and analysis workflows, such as pandas, is helpful
  • Understanding of RAG systems, retrieval, vector databases, LLM generation, and agent tool-calling workflows

Course Outline*:

*We customize the course outline and content to your specific needs and relevant use cases.

Module 1: Arize Phoenix and AI observability fundamentals

  • Understanding the role of tracing, evaluation, and experimentation in AI application development
  • Navigating Phoenix projects, traces, spans, attributes, and the observability interface
  • Understanding OpenTelemetry, OTLP, and OpenInference in the Phoenix tracing architecture
  • Running Phoenix locally or connecting applications to a hosted or self-managed Phoenix instance

Module 2: Instrumenting LLM applications

  • Instrumenting supported frameworks and model clients using OpenInference
  • Understanding automatic instrumentation and when manual spans are useful
  • Capturing prompts, model responses, retrieval operations, tool calls, and application metadata
  • Organising traces so that multi-step RAG and agent workflows remain understandable

Module 3: Trace analysis and application debugging

  • Following an individual request across nested spans and application components
  • Investigating errors, latency, token usage, model calls, and unexpected execution paths
  • Inspecting retrieval results and model context when debugging RAG responses
  • Analysing agent decisions, tool invocations, and multi-step execution behaviour

Module 4: RAG and agent observability patterns

  • Distinguishing retrieval, generation, tool, and orchestration problems within traces
  • Using metadata and attributes to compare services, models, environments, and application variants
  • Identifying repeated calls, unnecessary agent steps, poor retrieval, and expensive execution paths
  • Turning trace findings into concrete hypotheses for application improvement

Module 5: Evaluation fundamentals with Phoenix

  • Understanding the difference between deterministic, human, and LLM-based evaluations
  • Selecting evaluation criteria that reflect application quality and failure modes
  • Applying evaluators to spans, traces, and application outputs
  • Combining quality evaluations with operational information such as latency and cost

Module 6: Evaluating RAG systems and AI agents

  • Evaluating retrieval relevance and response faithfulness in RAG applications
  • Assessing answer correctness and other task-specific quality criteria
  • Evaluating agent tool selection, tool invocation, and handling of tool responses
  • Reviewing evaluator results and identifying false positives, false negatives, and unclear evaluation criteria

Module 7: Datasets and experiment analysis

  • Creating evaluation datasets from test cases, production traces, and reviewed examples
  • Defining inputs, expected outputs, metadata, and representative application scenarios
  • Running experiments to compare prompts, models, retrieval approaches, or application logic
  • Comparing experiment results across quality metrics, latency, cost, and meaningful data segments

Module 8: Debugging and continuous AI improvement

  • Moving from failed evaluations back to the underlying traces and application behaviour
  • Segmenting failures to identify patterns across queries, models, retrievals, or agent actions
  • Using experiments to validate fixes and detect regressions before deployment
  • Building a practical workflow connecting tracing, annotation, evaluation, experimentation, debugging, and application improvement

Hands-on learning with expert instructors at your location for organizations.

3,072€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
14
Hours (days:
2
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Immersive hands-on experience in a dedicated setting
*Price can range depending on number of participants, change of outline, location etc.

Master new skills guided by experienced instructors from anywhere.

2,037€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
14
Hours (days:
2
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Reduced training costs
*Price can range depending on number of participants, change of outline, location etc.

Upcoming Sessions

24-25 Feb 2027
Milan
31 Mar - 1 Apr 2027
London
29-30 Apr 2027
Lisbon
30 Jun - 1 Jul 2027
Madrid
8-9 Jul 2027
Frankfurt

Can't find a suitable date? Get in touch and we'll arrange one that works for you.

We use cookies to improve site navigation, analyse how the site is used, and support our marketing. You can accept all cookies, reject non-essential ones, or choose individual categories. Read our Cookie Notice