Spark

Provides automated optimization and performance analysis for PySpark code using MCP between a client and a Claude AI-powered server.

python

26

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "vgiri2015-ai-spark-mcp-server": {
      "command": "python",
      "args": [
        "v1/run_server.py"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "YOUR_API_KEY_PLACEHOLDER"
      }
    }
  }
}

You run an MCP server that coordinates AI-driven optimization of your Spark code. It connects with a client to analyze, optimize, validate, and report on performance, giving you automated, context-aware improvements to PySpark workloads.

How to use

Begin by preparing your Spark code for optimization. Start the MCP server to enable communication between the client and the server, then run the MCP client to submit your code for analysis and optimization. The system will analyze your code with Claude AI, generate an optimized version, validate it, and provide a performance analysis comparing the original and optimized versions. You can save the optimized code and the performance report for further review.

How to install

Prerequisites you need to have installed on your system are Python 3.8 or newer and PySpark 3.2.0 or newer.

Install required Python packages to run the MCP server and client.

Run the server and client as shown in the quick start steps.

pip install -r requirements.txt

# Start the MCP server
python v1/run_server.py

# In a separate terminal, start the MCP client
python v1/run_client.py

Notes and practical tips

Place your original Spark code in the specified input location before starting the client. The optimized code and a performance analysis will be generated in the output directory. The client and server collaborate to validate improvements, and the results are saved for reference.

Security and environment considerations

Use the ANTHROPIC_API_KEY environment variable to authorize Claude AI interactions. Keep this key secure and do not expose it in public code repositories.

Available tools

optimize_spark_code

Analyzes and generates an optimized PySpark code version with performance-oriented improvements. Saves the generated code to the designated output location and provides guidance on caching, shuffle partitions, and efficient expressions.

analyze_performance

Compares execution performance between the original and optimized Spark code, providing insights on timing, resource usage, and scalability.