analyze-video_skill

Overview

This skill adds visual descriptions to existing audio transcripts by extracting and analyzing video frames with ffmpeg. It creates a visual transcript JSON that pairs short scene descriptions with the corresponding transcript segments, without ever reading video files directly.

How this skill works

First, copy and prepare the audio transcript JSON to remove word-level timing and make it editable. Then extract a small set of JPG frames (start, middle, end, or additional samples when the scene changes) using ffmpeg. Read the extracted frames and incrementally edit the visual transcript JSON, inserting concise visual descriptions for dialogue and b-roll segments. Finally, clean up temporary frames and return a structured success message.

When to use it

• You already have an audio transcript for the video but no visual_transcript present.
• You need accessible visual descriptions for editors, producers, or accessibility workflows.
• You want short contextual scene annotations to guide video editing or scene selection.
• You are preparing content for platforms that require visual metadata alongside captions.

Best practices

• Always run the audio transcription first and work from the prepared visual_video.json file.
• Extract frames into a temporary directory and never read video files directly—use ffmpeg frame exports.
• Sample at most once per 30 seconds unless the footage clearly changes in subject, setting, or angle.
• Keep descriptions to three sentences max; make the first description detailed (subject, setting, shot type, lighting).
• Edit incrementally as you view frames rather than attempting to auto-generate everything in one pass.

Example use cases

• Add short visual descriptions to interview transcripts so editors can quickly locate cutaways and z-cuts.
• Generate visual metadata for long-form lectures where slides and camera framing change occasionally.
• Create accessibility-friendly transcripts for documentary clips with mixed dialogue and b-roll.
• Produce visual markers to guide automated or manual highlight extraction from event footage.