llm-monitoring-dashboard_skill

Overview

This skill automatically generates an LLM usage monitoring page for a PM-facing admin dashboard using the Tokuin CLI. It tracks tokens, cost, and latency per run, provides user ranking and non-user tracking, and produces data-driven PM insights. Built-in features include Cmd+K global search, per-user drilldowns, and multi-provider support (OpenAI, Anthropic, Gemini, OpenRouter).

How this skill works

The tool runs Tokuin load tests (dry-run by default) and collects JSON metrics into a local data store, enriching each record with user context, prompt category, and a prompt hash for privacy. A Next.js or lightweight Python frontend renders KPI cards, charts, ranking tables, and drill-down pages linked to each user or run. Scheduled cron jobs or manual collectors append metrics.jsonl and populate an SQLite dashboard for queries and alerts.

When to use it

• When you need real-time visibility into LLM API costs and token usage by team or individual
• When PMs require a weekly or ad-hoc admin dashboard to report who used the models and how
• When you want to identify and engage non-adopters to increase AI adoption
• When you need data to support model selection, cost optimization, or provider comparison
• When adding an LLM monitoring tab to an existing admin site with user drilldowns

Best practices

• Always run the included safety-guard before enabling live API calls to catch hardcoded keys, port conflicts, and large metric files
• Keep API keys out of code; store secrets in a .env that is gitignored
• Start in dry-run / estimate-cost mode to validate metrics and dashboards before allowing live calls
• Standardize user identifiers and aliases to make leaderboard and non-user tracking meaningful
• Set cost and latency thresholds and configure alerts (e.g., Slack webhook) to surface anomalies quickly
• Rotate or roll metrics files when size grows; import into SQLite for efficient querying and dashboards

Example use cases

• Weekly PM executive dashboard showing cost, latency percentiles, and top users by spend
• Detect sudden cost spikes or high-latency runs and drill into per-user runs to find root cause
• Compare models and providers in dry-run to decide a cost-optimized default model for a product flow
• Track adoption by flagging users with zero activity and launching onboarding nudges
• Schedule periodic benchmarks to monitor regression in latency or token usage after model or prompt changes