python-programming_skill

Overview

This skill provides practical Python programming guidance focused on fundamentals, data structures, and data science libraries like Pandas and NumPy. It helps you write efficient, maintainable Python code for data manipulation, analysis, and basic algorithm implementation. Use it to speed up workflows and avoid common performance and data-handling pitfalls.

How this skill works

The skill inspects common Python usage patterns and suggests idiomatic alternatives, emphasizing vectorized operations, correct data types, and library best practices. It guides file I/O with Pandas, missing-data handling, merging strategies, and performance tuning steps like chunking and dtype optimization. It also provides debugging checklists and concrete code snippets to apply immediately.

When to use it

• Writing or refactoring Python scripts for data cleaning and transformation
• Loading, merging, or exporting datasets with Pandas/NumPy
• Optimizing slow DataFrame operations and reducing memory usage
• Implementing algorithms that should leverage vectorization over loops
• Preparing data for machine learning or exploratory data analysis

Best practices

• Prefer vectorized NumPy/Pandas operations instead of Python loops
• Optimize DataFrame dtypes (e.g., int32, category) to save memory
• Profile before optimizing using %timeit and df.info() to target hotspots
• Handle missing values explicitly with fillna, dropna, or imputation strategies
• Use chunked reads for very large files and convert heavy string columns to category

Example use cases

• Read a large CSV in chunks, process each chunk, and write aggregated results
• Convert messy date and numeric columns to proper dtypes before analysis
• Merge multiple tables on keys with pd.merge and handle missing joins with how='left'
• Replace slow apply-based transformations with vectorized np.where or arithmetic on columns
• Diagnose memory issues by inspecting df.info() and converting repeated strings to category