jq-json-processing_skill

Overview

This skill provides practical commands and patterns for querying, filtering, and transforming JSON using the jq command-line tool. It focuses on extracting fields, manipulating arrays and objects, formatting output, and integrating jq into shell workflows. Use it to validate JSON, reshape API responses, and automate JSON-based tasks in scripts.

How this skill works

The skill exposes common jq expressions and pipelines to inspect and modify JSON structures. It covers basic accessors (.field, .[]), filters (select, map), object/array transforms (keys, sort_by, group_by), and output modes (-r, -c). It also shows advanced patterns like recursive descent, streaming large files, and try/catch for robust processing.

When to use it

• Parsing API responses and extracting specific fields for scripts or reports
• Filtering and aggregating large JSON arrays (logs, metrics, records)
• Transforming nested JSON into simplified objects for downstream tools
• Converting between JSON, CSV, and TSV formats for data exchange
• Validating JSON files and debugging unexpected structures

Best practices

• Start with simple queries and build complexity incrementally for easier debugging
• Use select() early to reduce data volume and improve performance on large files
• Prefer -r for raw string output in scripts and -c for compact JSON when speed matters
• Use .field? and // empty or try/catch to handle missing keys and avoid errors
• Break complex jq programs into steps or use variables (as $var) for clarity

Example use cases

• Extract all email addresses from a nested user dataset: jq '.. | .email? // empty' users.json
• Create a summary count by status: jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json
• Merge base and override configuration files: jq -s '.[0] * .[1]' base.json override.json
• Convert JSON records to CSV for spreadsheets: jq -r '.[] | [.name, .age, .email] | @csv' data.json
• Stream-process very large logs using --stream or process one JSON object per line with jq -c