Spark EventLog

You can run and connect to the Spark EventLog MCP Server to analyze Spark event logs, generate interactive HTML reports, and receive optimization recommendations. It supports local stdio execution, remote HTTP MCP access, and automatic data source detection from S3, HTTP URLs, or local files. This enables fast, end-to-end analysis and report generation across cloud and on-prem environments.

How to use

Connect an MCP client to the Spark EventLog MCP Server to start analysis, generate reports, and review optimization guidance. You can operate in stdio mode for local development, or in HTTP mode to expose a remote MCP endpoint that your clients can call over the network. Use the provided tools to trigger end-to-end report generation, monitor analysis status, and clear sessions as needed.

How to install

Prerequisites: make sure you have Python installed and a runtime capable of running the MCP server. You will also need a network path to the server if you plan to expose HTTP access.

# Install required dependencies (example workflow)
# Use your preferred environment setup; commands shown here are representative
uv install -e .

# Start the server in stdio mode (local development)
# The complete command shown in the example involves uv and a Python start script
MCP_TRANSPORT=stdio uv run python start.py

# Or start the server in HTTP mode for remote MCP access
export MCP_TRANSPORT=streamable-http
export MCP_HOST=0.0.0.0
export MCP_PORT=7799
uv run python start.py

Configuration and connection methods

The server exposes multiple MCP connection methods. Choose the one that best fits your deployment: an HTTP endpoint for remote clients, or a local stdio flow for development. The following configurations show the explicit ways to connect.

{
  "mcpServers": {
    "spark-eventlog": {
      "type": "http",
      "url": "http://localhost:7799/mcp",
      "args": []
    }
  }
}

{
  "mcpServers": {
    "spark-eventlog": {
      "command": "uv run python",
      "args": ["/path/to/spark-eventlog-mcp/start.py"],
      "env": {
        "MCP_TRANSPORT": "stdio"
      }
    }
  }
}

MCP Tools and usage

Access and use the built-in MCP tools to generate reports, check analysis status, and clear sessions. These tools are designed to be straightforward to call and to automate common workflows for log analysis and report generation.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "generate_report",
    "arguments": {
      "path": "s3://my-bucket/spark-logs/"
    }
  },
  "id": 1
}

Data sources

The server supports S3, HTTP URLs, and local files as input sources. It can auto-detect the data source type and proceed with log analysis and report generation.

{
  "source_type": "s3",
  "path": "s3://bucket-name/path/to/eventlogs/"
}

{
  "source_type": "url",
  "path": "https://example.com/eventlog.zip"
}

{
  "source_type": "local",
  "path": "/path/to/local/eventlog.zip"
}

Troubleshooting

If you encounter port conflicts, missing dependencies, or credential issues, follow these steps to resolve common problems.

# Change port
MCP_PORT=9090 python start.py

# Reinstall dependencies
uv pip install -e .

# Check AWS configuration
aws configure list

# Or set credentials in environment
AWS_ACCESS_KEY_ID=YOURKEY
AWS_SECRET_ACCESS_KEY=YOURSECRET

Notes and best practices

Leverage the modular architecture to tailor analysis, reports, and optimization recommendations to your Spark workloads. Enable enhanced logging to trace requests and responses for easier debugging and auditability.

Available tools

generate_report

End-to-end report generation in which the server auto-detects data sources, analyzes logs, and produces an HTML report

get_analysis_status

Query the current analysis session status and metrics

clear_session

Clear session cache and reset server state