tbench_skill

Overview

This skill provides Terminal-Bench integration for the Mux agent to run automated benchmarks and perform failure analysis. It bundles CI-friendly commands, Daytona cloud sandbox support, result publishing, and tools to investigate where Mux underperforms. Use it to run reproducible agent evaluations, collect logs, and prepare leaderboard submissions.

How this skill works

The integration invokes Terminal-Bench 2.0 tasks via a make-driven workflow that uses Harbor as the evaluation harness. It supports local Docker or Daytona cloud sandboxes for high-parallel runs, collects per-task artifacts and logs, uploads summarized results to BigQuery, and includes scripts to analyze failure rates and prepare leaderboard submissions. Configuration is driven by environment variables and TB_ARGS for model and agent kwargs.

When to use it

• Run full or partial benchmark suites in CI (workflow_dispatch) to validate agent versions.
• Speed up large runs using Daytona cloud sandboxes for high concurrency.
• Diagnose tasks where Mux fails more often than competitors using failure analysis scripts.
• Prepare and upload leaderboard submissions from successful nightly runs.
• Collect detailed per-task logs and asciinema sessions for debugging agent behavior.

Best practices

• Run benchmarks in CI with workflow_dispatch to ensure reproducible environment and artifacts.
• Use TB_CONCURRENCY and TB_ENV=daytona for faster results when available and within account limits.
• Prefer increasing the global TB_TIMEOUT if tasks consistently time out rather than per-task timeouts.
• Pass agent configuration via TB_ARGS --agent-kwarg (model_name, thinking_level, mode, experiments).
• Upload results and use BigQuery table mux-benchmarks.benchmarks.tbench_results for cross-run analysis.

Example use cases

• Run the full suite locally: make benchmark-terminal (default TB_CONCURRENCY=4).
• Run specific tasks: make benchmark-terminal TB_TASK_NAMES="hello-world chess-best-move".
• Run with a custom model and thinking level: TB_ARGS="--agent-kwarg model_name=openai/gpt-5-codex --agent-kwarg thinking_level=high" make benchmark-terminal.
• High-parallel Daytona run: export DAYTONA_API_KEY=... && TB_ENV=daytona TB_CONCURRENCY=48 make benchmark-terminal.
• Investigate failure hotspots: python benchmarks/terminal_bench/analyze_failure_rates.py --top 20 and download logs for failing runs.