python-performance_skill

Overview

This skill teaches practical Python performance optimization: profiling, bottleneck analysis, memory management, algorithm tuning, and high-performance libraries. It focuses on measurable improvements using profiling tools, vectorization, JIT/compiled extensions, and parallelism. Hands-on projects guide you from micro-benchmarks to production-ready optimizations.

How this skill works

The skill inspects code with profiling tools (timeit, cProfile, line_profiler, memory_profiler, py-spy) to locate CPU and memory hotspots. It teaches replacing slow patterns with efficient data structures, generator usage, NumPy vectorization, Numba/Cython acceleration, and multiprocessing or concurrent.futures for parallelism. You learn to benchmark changes, generate visual reports (flame graphs, HTML) and validate real speedups and memory reductions.

When to use it

• You need to reduce latency or throughput bottlenecks in production code
• Memory use grows uncontrolled or you suspect leaks
• A data pipeline or numeric workload is slow and can benefit from vectorization or JIT
• You must scale CPU-bound tasks across multiple cores or machines
• Before rewriting code in a lower-level language, to find the real hotspots

Best practices

• Profile before optimizing—measure first, change second
• Prefer algorithmic changes and better data structures over micro-optimizations
• Use generators and memory views to avoid unnecessary copies
• Apply NumPy/Numba/Cython only when profiling shows Python-level loops are the bottleneck
• Benchmark with representative data and repeat tests to avoid noise
• Limit global state and use context managers or object pools for deterministic cleanup

Example use cases

• Speed up a numeric kernel by replacing Python loops with NumPy or a Numba JIT function
• Reduce peak RAM of a CSV ETL job using streaming, generators, and chunked processing
• Identify and fix a memory leak using memory_profiler, gc, and weak references
• Parallelize a Monte Carlo simulation with ProcessPoolExecutor for near-linear speedup on CPU cores
• Build a profiling report with cProfile and flame graphs to justify architectural changes