altryne/weavify-skill

Overview

This skill adds W&B Weave observability and LLM tracing to any project to get token-level visibility, latency metrics, cost estimates, and debugging traces. It supports TypeScript/Node.js and Python with minimal code changes and automatic tracing for 20+ providers like OpenAI and Anthropic. Use it to instrument LLM calls, evaluate behavior, and monitor cost and performance.

How this skill works

Install the Weave client for your platform, set your WANDB_API_KEY, and call weave.init early in your app startup. Weave auto-patches supported LLM SDKs to capture inputs, outputs, token counts, latency, and cost. For unsupported or custom code paths you can wrap functions or clients with weave.op, wrapOpenAI, or manual wrappers to add traces.

When to use it

• You need token-level visibility for debugging or evaluation
• You want automatic latency and cost tracking for LLM calls
• Instrument a TypeScript/Node.js or Python project with minimal changes
• Add observability across many LLM providers without custom telemetry
• Verify and compare model responses and prompt performance over time

Best practices

• Call weave.init() early in application startup before any LLM calls
• Set WANDB_API_KEY in your environment securely and verify access to your W&B project
• Rely on auto-patching for supported SDKs; use weave.op or wrap helpers only when needed
• For ESM Node apps use --import=weave/instrument or mark LLM SDKs external in bundlers
• Add descriptive callDisplayName values to make traces easy to search and filter

Example use cases

• Add Weave to a chatbot to capture prompt/response tokens and latency for regression testing
• Wrap custom agent methods to surface internal reasoning and intermediate outputs in traces
• Compare model costs and latency across providers for a production routing layer
• Instrument CI runs to collect traces for evaluation datasets and automated QA
• Enable per-call cost dashboards to drive optimization and budget alerts