streaming-data_skill

Overview

This skill helps you design and implement production-ready event streaming and real-time data pipelines using Kafka, Pulsar, Redpanda, Flink, and Spark. It covers producer/consumer patterns, stream processing, event sourcing, and change data capture across TypeScript, Python, Go, and Java. The focus is on pragmatic guidance for delivery guarantees, error handling, and operational validation for reliable real-time systems.

How this skill works

I guide you through selecting the right message broker and stream processor based on latency, tenancy, and operational constraints, then show language-specific producer and consumer patterns. The skill lays out delivery guarantees (at-most-once, at-least-once, exactly-once), retry and DLQ strategies, and validation scripts to check broker connectivity and schema generation. It includes code examples and decision frameworks to build, test, and tune streaming pipelines end-to-end.

When to use it

• Building event-driven microservices or service-to-service communication
• Processing real-time analytics, monitoring, alerting, or dashboards
• Implementing CDC-driven data integration or cross-service synchronization
• Aggregating logs, metrics, or telemetry from distributed systems
• Developing low-latency platforms like IoT ingestion or trading systems

Best practices

• Choose broker and processor by latency, multi-tenancy, and operational complexity
• Default to at-least-once delivery and implement idempotent consumers unless exactly-once is required
• Use DLQs, exponential backoff retries, and graceful shutdown to avoid data loss
• Monitor consumer lag, error rates, throughput, and commit offsets deliberately
• Tune partitions, batch size, linger time, and compression to optimize throughput

Example use cases

• Event sourcing for domain state with immutable event stores and snapshots
• CDC pipelines using Debezium to stream DB changes into downstream services
• Real-time analytics using Flink for millisecond latency aggregations
• Embedded Kafka Streams in microservices for local stream transforms
• High-volume telemetry ingestion with Redpanda for low operational overhead