refactor-pandas_skill

Overview

This skill refactors Pandas code to improve maintainability, readability, and performance using modern Pandas 2.0+ features. It detects and replaces slow patterns (loops, .iterrows(), heavy .apply()), fixes chained indexing and SettingWithCopyWarning issues, and applies dtype and pipeline optimizations. The goal is safe, testable refactors that preserve results while reducing runtime and memory usage.

How this skill works

The refactor inspects code for anti-patterns: row-wise loops, unnecessary .apply() calls, chained indexing, inplace=True usage, inefficient dtypes, missing method chaining, complex boolean masks, and unsafe merges. It suggests and produces vectorized replacements, method-chained pipelines (.pipe/.query/.eval/.assign), dtype conversions (categories, downcasting, PyArrow backend), and merge validations. Outputs include change summaries, before/after snippets, and performance impact notes.

When to use it

• When code iterates over DataFrame rows (.iterrows, .itertuples, or for loops) for column computations
• When .apply() is used for numeric or elementwise operations that can be vectorized
• When SettingWithCopyWarning or chained indexing appears in notebooks or production code
• When memory use is high and dtypes can be optimized (categorical, downcast, PyArrow)
• When complex filters are hard to read and would benefit from .query()/.eval()
• Before merging datasets to ensure validation and avoid accidental duplications

Best practices

• Prefer vectorized ops and NumPy where possible; reserve .apply() for genuinely row-wise logic
• Compose transformations with method chaining and .pipe() to keep single-responsibility steps
• Enable copy-on-write and consider dtype_backend='pyarrow' for large datasets
• Use .loc for assignments and explicit .copy() to avoid SettingWithCopyWarning
• Always specify merge parameters (how, on, validate) and add an indicator for debugging
• Write small, testable functions with input validation and use pd.testing.assert_frame_equal after refactor

Example use cases

• Replace a loop that computes new columns with vectorized arithmetic and .loc assignments
• Convert status/text columns to category to cut memory and speed up groupby
• Refactor chained filters into a readable .query() pipeline and collapse intermediate variables
• Add dtype optimization and PyArrow conversion to reduce memory for large CSV loads
• Replace ambiguous inplace mutations with explicit assignments under copy-on-write