altinity-expert-clickhouse-part-log_skill

Overview

This skill diagnoses ClickHouse cluster issues by analyzing system.part_log entries (part creation, merges, mutations, downloads, removals, moves). It highlights top offenders by event rate, volume, and error counts to quickly surface ingestion, merge, mutation, and replication hotspots. Use it to decide which focused diagnostic or remediation module to run next.

How this skill works

The skill runs a curated set of time-bounded SQL checks against system.part_log (cluster-aware) and ranks results by events/min, rows/bytes moved, and error occurrences. Queries default to recent relative windows (last 1h/6h/24h) and adapt if schema differences are detected by inspecting system.part_log. It maps discovered patterns to follow-up diagnostic modules for deeper analysis.

When to use it

• When you see too many small parts or suspect micro-batch inserts
• When merges are backlogged or merge throughput is unexpectedly low
• When ALTER DELETE/UPDATE activity spikes and mutations are frequent
• When replication shows high DownloadPart churn or repeated downloads
• When unexpected RemovePart or MovePart spikes coincide with znode growth

Best practices

• Always run checks with relative time windows (default last 1h/6h/24h) unless a specific range is requested
• Replace {cluster} with your ClickHouse cluster identifier and run cluster-wide where appropriate
• Limit results and keep queries time-bounded to avoid heavy system load
• If a query fails, DESCRIBE TABLE system.part_log and drop only the missing fields before re-running
• Follow cross-module triggers: map high-rate events to the corresponding expert module for deeper diagnostics

Example use cases

• Detecting micro-batch insert patterns causing high NewPart rates and planning batching or KV buffering
• Identifying tables with slow or saturated merges to tune merge settings or storage layout
• Tracing mutation storms from frequent ALTER DELETE/UPDATE and deciding to rewrite or re-partition data
• Spotting replication instability by finding tables with repeated DownloadPart activity and checksum errors
• Correlating RemovePart spikes with Keeper/ZooKeeper znode growth to diagnose replication metadata churn