ontos-ai/skills-evaluator

Overview

This skill is a meta-evaluator that inspects and scores Claude Skills for quality and readiness. It runs progressive checks from fast static analysis to optional LLM-driven smoke tests, then produces a structured JSON report with issues and actionable recommendations. The goal is to surface structural problems, vague instructions, and missing integrations so maintainers can quickly fix and publish reliable skills.

How this skill works

The evaluator performs layered checks: Level 1 runs static analysis on the skill files and directory layout; Level 2 runs optional smoke tests that invoke an LLM to validate prompts and example behavior. It measures five dimensions—structure, triggers, actionability, tool integration, and examples—and computes per-dimension scores and an overall badge. The output is a machine-readable JSON report and optional Markdown summaries for CI or review.

When to use it

• Before publishing a new skill to catch structural and content defects
• During CI to block low-quality or malformed skill submissions
• When adding LLM-driven behavior to verify prompts and example outputs
• To audit a collection of skills for consistency and integration issues
• When you need a prioritized list of actionable fixes and severity-tagged issues

Best practices

• Run Level 1 (static) checks early in development to catch syntax and metadata errors
• Use Level 2 smoke tests with a configured LLM provider to validate runtime prompts and expected outputs
• Include concrete, realistic examples and explicit trigger phrases in the skill documentation
• Reference actual scripts, assets, and docs with correct relative paths to avoid broken links
• Use the JSON report in CI to programmatically gate releases and generate human-readable change tickets

Example use cases

• Quick local audit of a single skill before release (interactive or scripted)
• Batch-evaluate a directory of skills and produce a consolidated JSON report for governance
• Run smoke tests against multiple LLM providers to compare behaviour and robustness
• Integrate into a skill generation pipeline to validate outputs before distribution
• Generate Markdown compliance reports for QA reviews or developer handoff