osworld-observe_skill

Overview

This skill captures the current observation from an OSWorld environment and returns both a screenshot (base64 PNG) and the raw accessibility tree (JSON). It provides a timestamped, unfiltered snapshot of the UI state for automated agents to inspect, store, or process downstream. Configuration is available via an environment variable or character config.

How this skill works

When invoked, the skill requests an observation from OSWorld and packages the response into a note ID bound to the configured output variable. The observation includes a base64-encoded PNG screenshot and the raw accessibility tree JSON, along with metadata such as timestamp and step counter. Screenshot and accessibility data can be included or omitted using boolean flags.

When to use it

• Capture a visual and accessibility snapshot before or after an automated action.
• Gather raw UI state for debugging, test verification, or training data collection.
• Archive reproducible observations for audit trails or issue reports.
• Compare pre- and post-action UI state to validate effect of automation steps.
• Feed the raw accessibility tree into downstream analysis or accessibility checks.

Best practices

• Enable only the data you need: set include_screenshot or include_a11y to false to reduce payload size.
• Store the returned base64 PNG and JSON separately to simplify downstream processing.
• Use the timestamp and step_counter metadata to reliably correlate observations with actions.
• Avoid interpreting or modifying the raw accessibility tree inside this step; do analysis in dedicated processing steps.
• Keep OSWORLD_URL configured consistently across environments to avoid pointing to the wrong instance.

Example use cases

• Capture screenshot + accessibility tree immediately after launching an app to verify initial layout.
• Take observations before and after a click to confirm UI changes and generate a diff.
• Collect accessibility trees across screens to build a map of interactive elements for automated testing.
• Produce a visual and structural record for bug reports, attaching the base64 PNG and raw JSON.
• Use observations as labeled inputs for training models that predict next actions from UI state.