f5tts-monitor_skill

Overview

This skill monitors the F5-TTS distributed training running on the 9-GPU mining rig (Local-LLM) and reports status without altering data or interfering with the process. It is designed for safe, read-only probes that collect GPU, epoch, and system health information. The monitor follows strict constraints to avoid running training locally or modifying remote checkpoints.

How this skill works

The monitor connects to the mining rig via SSH and runs read-only commands to inspect GPU utilization, training log progress, and system memory/CPU load. It parses the Accelerate training log to extract current epoch and global step, samples nvidia-smi output to confirm per-GPU VRAM usage, and checks system free memory and load averages. After probing, it updates a local heartbeat report with epoch, step, GPU temperature, and an estimated time remaining.

When to use it

• To verify all 9 GPUs are actively participating in distributed training and not OOMing.
• To confirm training progress by reading the Accelerate training log for epoch and global step.
• To ensure the mining rig’s CPU and RAM are not overloaded by DDP overhead.
• Before reporting status updates to the project owner or automated dashboards.
• When you need a non-invasive status snapshot without restarting or altering processes.

Best practices

• Always connect to the mining rig via SSH and use a pseudo-terminal when running live watch-style probes.
• Use the designated Python environment alias ('uv') on the rig for any environment-specific inspections.
• Never initiate or attempt to run training on the local workstation (asus-z170k).
• Treat all probes as read-only: do not move, edit, or copy dataset files or checkpoints on the rig.
• Update HEARTBEAT.md locally with concise fields: Epoch, Step, GPU temps, and ETA.

Example use cases

• Quick health check to confirm nine python3 training processes are each using ~11GB VRAM.
• Tail the training log to extract current epoch/step before sending a status update to the team.
• Validate system free memory and load averages after adding monitoring services to avoid overload.
• Generate a heartbeat report for Master Seiya with epoch, step, GPU temperature, and ETA.
• Automated cron probe that collects nvidia-smi and log snippets for archival and alerting.