phoenix_skill

Overview

This skill is an open-source AI observability platform for tracing, evaluating, and monitoring LLM applications. It provides end-to-end traces, LLM-as-judge evaluators, dataset/versioning tools, experiments, and a real-time UI for production monitoring. Use it to debug models, run systematic evaluations, and self-host observability without vendor lock-in.

How this skill works

Phoenix instruments LLM frameworks via OpenTelemetry to capture traces and spans for each model call. It runs automated evaluations using built-in or custom LLM evaluators, stores versioned datasets and examples, and exposes APIs and a web UI for querying traces, runs, and metrics. A lightweight client lets you export data, log annotations, and run experiments or evaluation pipelines programmatically.

When to use it

• Debug unexpected LLM behavior with detailed request/response traces
• Run automated evaluations or quality checks on model outputs and datasets
• Monitor production LLM systems in real time for latency, errors, and token usage
• Build experiment pipelines to compare prompts, models, or configurations
• Self-host observability to avoid vendor lock-in and control data storage

Best practices

• Separate traces by project/environment (dev/staging/prod) to reduce noise
• Attach metadata (user IDs, session IDs, request ids) to spans for faster debugging
• Version datasets and add examples for regression testing and reproducibility
• Run evaluations regularly in CI/CD and log results back to Phoenix
• Use PostgreSQL in production and monitor DB connectivity and resource usage

Example use cases

• Instrument OpenAI, Anthropic, LangChain, or LlamaIndex to capture end-to-end traces during chat or retrieval flows
• Run hallucination and relevance evaluators on a set of traces and log aggregated metrics
• Create a QA dataset, run experiments comparing two models, and store per-example evaluation labels
• Deploy Phoenix in Docker with PostgreSQL to monitor token usage and latency in production
• Use the client API to export spans to pandas and run offline analysis or custom reports