data-profiler_skill

Overview

This skill profiles construction datasets to surface column-level characteristics, distributions, quality metrics, and common patterns. It summarizes null rates, uniqueness, top values, numeric statistics, detected patterns (like CSI codes or currency), and an overall quality score. The output includes actionable recommendations for ETL, data cleaning, and downstream analyses. Use it to catch data issues before they impact reporting or project workflows.

How this skill works

The profiler inspects each column to compute counts, null percentages, unique counts, top values, and type hints inferred from column names and value patterns. Numeric columns receive min/max/mean/median/std checks and outlier detection; string columns are scanned for length statistics and pattern matches (project IDs, CSI codes, dates, currency, etc.). It aggregates column scores into an overall quality score and generates targeted recommendations such as converting types, cleaning currency fields, or reviewing high-null columns.

When to use it

• Before importing or transforming construction data into an ETL pipeline
• During data quality assessments for project budgets, cost tracking, or schedules
• When onboarding new data sources or merging vendor feeds
• Prior to analytics or dashboard creation to avoid misleading metrics
• To validate expected formats for construction-specific fields (CSI, WBS, project IDs)

Best practices

• Run the profiler on a representative sample and the full dataset to compare results
• Use inferred type hints and pattern matches to automate type conversions (dates, currency)
• Investigate columns flagged with high null rates or many quality issues first
• Normalize identifier and code formats (project_id, csi_code) early in ETL
• Record profiler output (JSON or report) as a data-contract artifact for downstream teams

Example use cases

• Audit a subcontractor CSV feed to find inconsistent project IDs and currency formatting before ingestion
• Scan historical cost tables to identify negative values, outliers, and low-cardinality category fields
• Validate schedule exports for date format issues and convert date strings to datetime
• Profile equipment or material lists to detect duplicate rows and high-null attribute columns
• Generate a quality report to prioritize cleaning tasks before building cost dashboards