data-analysis-patterns_skill

Overview

This skill provides practical patterns and decision rules for data aggregation, recalculation, category management, and multi-source merging in scientific analyses. It focuses on when to recalculate from raw records versus reuse aggregates, how to separate conflated features, and robust strategies for composite keys and AWS enrichment. The guidance is aimed at producing accurate, publication-ready results while enabling pragmatic shortcuts for exploration.

How this skill works

The skill inspects aggregated tables, raw-level records, and enrichment sources to determine whether existing aggregates can be remapped or must be recomputed. It provides patterns for creating composite keys, resolving duplicate records, validating external paths (e.g., S3), and separating conflated features into independent analyses. It also codifies conflict-detection and resolution strategies when merging updates.

When to use it

• Deciding whether to recalculate aggregates after category definition changes
• Merging datasets from multiple sources where single keys are not unique
• Preparing publication-quality figures that require exact counts
• Separating metrics that were previously conflated (e.g., two features combined in one category)
• Enriching tables with external data (AWS S3) while preserving manual edits

Best practices

• Recalculate from raw data whenever category definitions fundamentally change or mapping would be lossy
• Create a multi-part composite key (most specific → least specific) and verify uniqueness immediately
• Resolve duplicate composite keys by latest curated date or by record completeness
• Document any approximations clearly and restrict them to exploratory analyses, not final figures
• Validate inferred external resources (S3 paths) before attempting bulk fetches and use temporary columns with underscore prefixes

Example use cases

• Recompute species-level percentages from scaffold-level raw data after redefining category rules
• Build a 3-part composite key (ToLID|Assembly|Pipeline) to merge Google Sheets with AWS-enriched QC tables
• Separate a conflated telomere category into independent terminal-count and interstitial analyses for clear interpretation
• Infer and validate missing S3 paths in a target table using two-tier resolution (direct lookup then inferred+validate)
• Run conflict detection when merging updated base metadata and let user choose NEW vs OLD values for resolution