vom-algorithms_skill

Overview

This skill implements and extends the Visual Object Model (VOM) algorithms for detecting UI elements on terminal screens. It provides segmentation, clustering, heuristic classification, and component tracking so agents can reason about terminal layouts and target UI elements reliably. Use it when changing detection logic, adding roles, or optimizing incremental updates in an agent-driven TUI pipeline.

How this skill works

The pipeline treats the terminal as a 2D ScreenBuffer of styled cells, performs run-length segmentation into style-homogeneous Clusters, groups clusters into spatial regions, and applies heuristic classifiers to emit Components with roles and visual hashes. It supports incremental update paths to match components across frames and hit-testing for click targeting.

When to use it

• You are modifying segmentation or run-length encoding predicates.
• You need to add or refine UI element roles or heuristics (e.g., progress bars, buttons).
• Implementing incremental updates or performance optimizations for snapshots.
• Debugging detection failures related to styles, coordinates, or Unicode cell widths.
• Adding click targeting, hit-testing, or multi-row component detection.

Best practices

• Read the segmentation and pipeline design docs before changing core logic to avoid regressions in traversal order and complexity.
• Keep segmentation O(W×H) by preferring single-pass predicates and avoid expensive per-cell computations.
• Add new roles as low-cost heuristic checks and insert them in priority order inside infer_role to preserve cascade behavior.
• Write unit tests for positive, negative, boundary, and style-variation cases (edge-of-screen and empty text).
• Use content hashing to track elements across frames and implement incremental diffing to limit reclassification work.

Example use cases

• Add a ProgressBar role: extend Role enum, implement is_progress_bar(), and add tests that assert detection on block/box characters.
• Change merge logic in segmentation to allow grouping by style subsets (e.g., ignore color differences) by adjusting should_merge().
• Improve performance by implementing incremental-updates: compute visual_hash, find matching components in new frame, and only reclassify changed regions.
• Add wide/emoji support by applying Unicode cell width handling during clustering and bounding-box computation.
• Implement click targeting by computing component bounding boxes and mapping screen coordinates to component roles.