mixseek-evaluator-config_skill

Overview

This skill generates MixSeek evaluation configuration files (evaluator.toml and judgment.toml) to define submission scoring, metric weights, and final judgment logic for TUMIX tournaments. It guides metric selection, weight rules, and produces validated TOML files placed under the workspace config paths. Use it to produce reproducible, verifiable evaluation rules for automated scoring pipelines.

How this skill works

The skill asks targeted questions about evaluation focus, weighting, and judgment style, then proposes metric sets and weight distributions. It outputs evaluator.toml and judgment.toml content following MixSeek conventions, enforces weight-sum and metric-name rules, and provides validation commands to run after generation. It also warns about custom paths and orchestrator configuration requirements.

When to use it

• Creating or updating submission scoring rules for a tournament
• Defining metric weights and final decision logic for automated evaluation
• Switching between deterministic and diversity-aware judgment styles
• Generating evaluator and judgment TOML files for CI or workspace deployment
• Validating generated configuration before running large-scale evaluations

Best practices

• Decide evaluation priority (clarity, coverage, relevance) before choosing metrics
• Either specify weights for all metrics or omit all to use equal distribution
• Ensure total weight sums to 1.0 (±0.001) when using custom weights
• Set judgment temperature to 0.0 for deterministic decisions and fix seed for reproducibility
• If using custom config paths, declare them in the orchestrator config to ensure discovery

Example use cases

• Generate equal-weight evaluator for general-purpose answer scoring (ClarityCoherence, Coverage, Relevance)
• Create a relevance-focused evaluator (Relevance 0.5, ClarityCoherence 0.3, Coverage 0.2) for factual QA tasks
• Define an LLMPlain metric with a system_instruction for domain-specific evaluations (security, legal, medical)
• Produce judgment.toml with deterministic settings (temperature=0.0, timeout_seconds configured) for final pass decisions
• Validate generated TOML files in CI before deploying scoring jobs