data-lineage-tracker_skill

Overview

This skill tracks the origin, transformations, and flow of construction data across systems to produce reliable audit trails and support compliance. It captures sources, entities, transformation steps, checksums, and lineage records so teams can trace where values came from and how they were derived. The tracker is designed for inspection, debugging, and impact analysis in construction projects.

How this skill works

The tracker registers data sources and entities, records transformation steps with metadata (type, logic, actor, timestamp), and creates lineage records linking upstream and downstream entities. It exposes trace functions to walk upstream and downstream dependencies, generate entity history, calculate impact of changes, validate consistency, and render a Mermaid-style lineage graph. Checksums and versioning help verify data integrity and detect tampering or drift.

When to use it

• Establish audit trails for bids, estimates, and regulatory reporting
• Debug unexpected values by tracing data back to its origin
• Perform impact analysis before changing a dataset or ETL step
• Validate completeness and detect broken or circular lineage links
• Document provenance for insurance, legal, or compliance reviews

Best practices

• Register every source and entity as soon as it enters a workflow to avoid orphan records
• Record transformation logic and parameters (SQL, scripts, manual edits) for reproducibility
• Use checksums or content hashes on snapshots to detect accidental changes
• Limit trace depth for performance and surface high-priority upstream/downstream issues first
• Regularly run validate_lineage to catch broken references and circular dependencies early

Example use cases

• Trace a cost discrepancy from a project summary line back to the original field report and sensor import
• Assess which downstream reports and dashboards will break if a supplier invoice feed changes format
• Generate a compliance-ready provenance report showing every transformation applied to safety inspection data
• Find orphaned data outputs that were created by ad-hoc exports and never linked into ETL pipelines
• Visualize data flow for stakeholder review using generated Mermaid diagrams