apache-spark-data-processing_skill

Overview

This skill is a complete guide to Apache Spark data processing covering RDDs, DataFrames, Spark SQL, streaming primitives, MLlib, and production deployment patterns. It focuses on building scalable distributed pipelines, optimizing queries, and applying machine learning at scale. The content is practical, with examples for transformations, caching, partitioning, and common production concerns.

How this skill works

The guide explains core Spark abstractions (RDD, DataFrame, Dataset) and how lazy evaluation builds a DAG that Spark optimizes and executes across a cluster. It covers reading/writing common formats (Parquet, ORC, JSON, CSV), Spark SQL features (Catalyst, AQE), and shared-variable patterns (broadcasts and accumulators). It also describes caching strategies, partition tuning, UDF usage, and window functions for advanced analytics.

When to use it

• Process datasets larger than a single machine (TB+).
• Run parallel distributed computations and ETL/ELT pipelines.
• Combine batch and streaming workloads or handle real-time streams.
• Train or serve machine learning models on large data using MLlib.
• Perform complex aggregations, joins, and SQL analytics at scale.

Best practices

• Prefer DataFrames/Datasets over low-level RDDs for performance and optimizer benefits.
• Tune partition count: aim for 2–4 tasks per CPU core; use repartition/coalesce carefully.
• Cache only data reused multiple times; choose MEMORY_AND_DISK for large intermediate sets.
• Use built-in Spark functions or Pandas UDFs instead of Python UDFs to avoid serialization overhead.
• Broadcast small lookup tables; use accumulators for driver-side aggregated counters.

Example use cases

• Large-scale ETL: ingest JSON/CSV, transform with DataFrame API, write partitioned Parquet for analytics.
• Streaming analytics: process Kafka streams with Structured Streaming and maintain aggregations or stateful windows.
• ML at scale: preprocess features with DataFrame ops, train MLlib models, and persist models to a shared store.
• Ad-hoc interactive analysis: explore petabyte-scale datasets via Spark SQL and cached intermediate results.
• Complex joins and windowed reports: compute ranked leaderboards and running totals with window functions.