pueue-job-orchestration_skill

Overview

This skill provides a universal CLI telemetry layer and job orchestration wrapper around Pueue for reliable, observable execution of shell commands across local and remote hosts. It captures timing, exit codes, full stdout/stderr logs, environment snapshots, and supports callbacks, delays, and priority handling. Use it to queue, monitor, and manage long-running, batch, or GPU-bound tasks with persistence across SSH disconnects and reboots.

How this skill works

Commands routed through the skill are submitted to Pueue as tasks with labels, groups, and optional dependency wiring using --after. The orchestration records task metadata (start/end times, exit codes, env snapshots) and stores logs on disk so jobs auto-resume after failures. Group-based parallelism, per-task callbacks, and priority/delay flags let you control concurrency and ordering on hosts like BigBlack, LittleBlack, or local macOS.

When to use it

• Any command expected to run longer than ~30 seconds
• Batch operations with multiple items or symbols (>3)
• Build/test or multi-stage pipelines that need dependency chaining
• Remote execution over SSH where persistence is required
• Cache population, heavy data processing, or GPU workstation jobs
• When you need full telemetry, logs, and simple failure recovery

Best practices

• Always run durable jobs through pueue to get logs and checkpointing; reserve direct SSH for quick one-offs
• Use groups to limit concurrent work per resource class and per-symbol to avoid contention
• Capture job IDs with --print-task-id and wire post-processing via --after instead of manual polling
• Split large multi-year workloads into epoch/year shards to gain parallelism and safer checkpoints
• Audit remote host state (pueue status, system resources, checkpoints) before mutating production data

Example use cases

• Queue a long python data pipeline on BigBlack and follow output remotely
• Populate per-symbol cache with per-year shards using group parallelism
• Chain batch processing -> OPTIMIZE -> validation by capturing job IDs and using --after
• Install and start pueued on a Linux GPU server and submit nightly batch jobs with callbacks
• Restart all failed jobs after a transient issue with pueue restart --all-failed