model-matrix_skill

Overview

This skill provides a weighted model-routing matrix that selects the best model per category using blended scores and policy constraints. It combines task evaluations, benchmark results, sentiment signals, and cost to produce an effective routing decision. The skill includes a daily scorecard template and operational rules to ensure stable, cost-aware routing choices.

How this skill works

The system computes a raw score for each candidate model per category by applying predefined weights: real task evaluations (45%), benchmarks (30%), sentiment signals (20%), and cost (5%). Policy rules then filter and adjust rankings—preferring cheaper models that preserve quality, auto-promoting alternatives if specific providers are excluded, and only swapping routes when score deltas are material and confidence is high. The output is an effective model choice per category plus a confidence indicator and provenance fields for auditing.

When to use it

• When you need a consistent, auditable method to pick models across functional categories
• When cost and provider-exclusion policies must influence routing decisions
• For daily monitoring and automated policy-driven updates to model routes
• When blending real-world evals, benchmarks, and public sentiment matters
• When you want an operational scorecard to guide on-call or ops decisions

Best practices

• Keep evaluation datasets stable and representative to avoid noisy score shifts
• Treat sentiment signals as directional; require high confidence before route changes
• Enforce the core policy: prefer the cheapest model that preserves quality
• Auto-promote alternatives only when exclusions are explicit and documented
• Run trial periods (e.g., 7 days) before committing to new minis or experimental models

Example use cases

• Daily ops scorecard that recommends the effective model for research, coding, and creative tasks
• Automated router that demotes expensive options if cheaper alternatives match quality thresholds
• Policy-driven failover when a vendor is excluded or temporarily unavailable
• Trend-monitoring for citizen sentiment to adjust routing for social listening tasks
• A/B trial management where models marked trial-only are promoted only after a successful window