arize-ai/phoenix

Overview

This skill lets you debug and analyze LLM applications using the Phoenix CLI (px). It provides commands to fetch traces, inspect spans, review experiments and datasets, and extract LLM usage metrics like model names and token counts. Use it to speed up root-cause analysis and to surface performance or failure patterns in observability data.

How this skill works

The skill wraps common px CLI workflows: list and filter traces, fetch detailed trace and span data, inspect datasets and experiments, and emit machine-friendly raw JSON for further processing. It encourages piping raw output to tools like jq for custom analyses, and supports multiple output formats (pretty, json, raw) and file export options. Environment variables configure host, project, and API key, while CLI flags can override them.

When to use it

• Investigating failed runs or exceptions reported by an LLM-based service
• Profiling latency or finding the slowest traces and spans
• Extracting model usage, token counts, and cost-related metrics
• Reviewing experiment results and dataset example failures
• Exporting traces or experiments for offline analysis or reporting

Best practices

• Use --format raw --no-progress when piping to jq or other CLI tools to avoid formatting issues
• Start with px traces --limit N to find candidate traces, then fetch details with px trace <id>
• Filter spans by span_kind (LLM, CHAIN, TOOL, etc.) to narrow troubleshooting scope
• Record the px command and filters you used to reproduce findings and share with teammates
• Set PHOENIX_HOST, PHOENIX_PROJECT, and PHOENIX_API_KEY as env vars for reproducible commands

Example use cases

• Fetch the most recent 50 traces and list those with status == ERROR to identify failures
• Sort traces by duration and inspect the top 5 slow traces to pinpoint performance bottlenecks
• Extract llm.model_name and token counts across traces to audit model usage and token consumption
• List experiments for a dataset and compute average latency or extract items with non-null errors
• Export a dataset split to a file with px dataset <name> --split validation --file out.json