kafka_skill

Overview

This skill provides a tested, opinionated workflow to deploy and operate Apache Kafka on Kubernetes using the Strimzi operator in KRaft mode (no ZooKeeper). It includes manifests, a Makefile for one-command deployment, and Python producer/consumer templates for quick integration. The skill adapts resource profiles to target environments and includes runbooks for common debugging scenarios. Use it to deploy clusters, create topics, build clients, and diagnose streaming issues.

How this skill works

Before generating or applying manifests the skill detects host and cluster resources (memory and CPU) and selects an adaptive resource profile (Minimal, Standard, Production). It installs Strimzi via Helm, applies KRaft-mode Kafka CRs and topic CRDs, and exposes bootstrap endpoints for local and in-cluster access. The package contains Python templates for producers/consumers, runbooks for debugging (lag, rebalances, topic inspection), and a Makefile with install/test/status/uninstall targets for repeatable workflows.

When to use it

• Deploy a Kafka cluster on Kubernetes where ZooKeeper is not desired (KRaft).
• Rapid local development with Docker Desktop or a small k8s cluster and need working producer/consumer examples.
• Implement event-driven patterns, domain events, or CloudEvents with schema guidance.
• Debug consumer lag, topic metadata, or broker/controller issues in KRaft clusters.
• Generate manifests adapted to the available node/container memory to avoid resource overcommit.

Best practices

• Always run the resource detection step and choose the profile matching your environment.
• Prefer the Standard profile for general workloads; use Production only on real clusters with >24GB.
• Use acks=all and enable.idempotence for at-least-once guarantees; use transactions for exactly-once where required.
• Expose bootstrap.servers appropriately: localhost:30092 for host dev, cluster service DNS for pod-to-pod.
• Automate install/test/status/uninstall via the provided Makefile for reproducible deployments.

Example use cases

• One-command deploy of Strimzi + KRaft Kafka for a staging environment using make install.
• Generate adaptive manifests for a low-RAM developer laptop and warn about constraints.
• Build a FastAPI-based async consumer using the provided Python template for event-driven microservices.
• Run debugging runbooks to identify consumer lag and describe topic partitions during an incident.
• Create KafkaTopic CRDs to provision topics as part of CI/CD pipelines.