Parquet

You manage parquet-based data with an MCP server that provides query, insert, update, upsert, delete operations, plus a robust audit trail, rollback, schema discovery, and semantic search. This server is designed to minimize storage usage for audit history while offering optional periodic full backups for extra safety, making it practical for reliable data governance and efficient recovery.

How to use

You connect to this MCP server using an MCP client. Start by configuring a local or development environment to run the Python MCP server, then issue requests to manage parquet data types such as flows, transactions, tasks, or contacts. You can perform reads with filters and column selection, add new records, update or delete existing ones, and execute upsert operations that create a record when none exists or update when a match is found. All modifications create an audit log entry to enable rollbacks and history tracking. When you enable full snapshots, you receive periodic backups in addition to the audit trail.

What you can do with the server

• Read and query parquet data with optional filters, selecting specific columns and limiting results.

• Add new records to parquet files. Each add creates an audit log entry.

• Update existing records that match filters. Each update creates an audit log entry.

• Upsert records: insert if not found or update if a match exists. This can prevent duplicates and returns whether a record was created or updated along with audit IDs.

• Delete records that match filters. Each delete creates an audit log entry.

• Review the audit log to see complete history and old/new values for modifications.

• Rollback a specific operation by audit ID to undo changes. The rollback creates the inverse operation to restore prior state.

• Discover schemas for data types and obtain basic statistics about parquet files.

• Generate embeddings for semantic search and perform search across text fields using embeddings and similarity metrics.

How to install

Prerequisites: ensure you have Python installed on your system.

cd mcp-servers/parquet
pip install -r requirements.txt

Production vs development configuration

Development (Audit Log Only): in your Cursor settings, run the MCP server with audit logging enabled and no full snapshots.

{
  "mcpServers": {
    "parquet": {
      "command": "python",
      "args": [
        "$REPO_ROOT/mcp-servers/parquet/parquet_mcp_server.py"
      ],
      "env": {}
    }
  }
}

Production (With Periodic Snapshots)

In production, enable periodic full snapshots to provide additional safety while using the audit log for efficient storage.

{
  "mcpServers": {
    "parquet": {
      "command": "python",
      "args": [
        "$REPO_ROOT/mcp-servers/parquet/parquet_mcp_server.py"
      ],
      "env": {
        "MCP_FULL_SNAPSHOTS": "true",
        "MCP_SNAPSHOT_FREQUENCY": "weekly"
      }
    }
  }
}

Claude Desktop configuration

Add the same Python-based MCP server configuration to Claude Desktop so the tool can connect to the parquet MCP server.

{
  "mcpServers": {
    "parquet": {
      "command": "python",
      "args": [
        "$REPO_ROOT/mcp-servers/parquet/parquet_mcp_server.py"
      ]
    }
  }
}

Backup & Recovery

Audit logs provide lightweight change history for each write operation, usually around 1 KB per operation, reducing storage needs by a large margin compared with full snapshots.

Optional full snapshots can be configured for extra safety and can be stored under data/snapshots with a timestamped filename.

If you need to recover, you can rollback a specific operation using its audit ID, or restore from a full snapshot and replay the audit log up to the desired point in time.

Data types

The server automatically discovers data types by scanning data/ for parquet files. Common data types include flows, transactions, tasks, and contacts, with many others supported.

Security Notes

All write operations create audit log entries for traceability. Audit logs are stored in data/logs/audit_log.parquet and optional full snapshots can be configured for additional safety. Do not commit sensitive data files to version control.

Troubleshooting

If you encounter file not found errors, verify the data type exists under data/[type]/[type].parquet and check permissions. For schema validation errors, ensure records match the schema defined under data/schemas/[type]_schema.json and that required fields are present. If filter matching errors occur, confirm filter syntax and column availability.

Testing

Run the test script to validate audit log creation, schema compliance, and operation tracking after installation or updates.

python3 mcp-servers/parquet/test_audit_log.py

Notes

The server uses an audit log for efficient change tracking, converts date fields to ISO format in responses, and represents null/NaN values as null in JSON responses. The MCP server runs in stdio mode for MCP communication, and audit log entries are never automatically deleted.

Available tools

list_data_types

List all available data types (parquet files) in the data directory.

get_schema

Get the schema definition for a data type.

read_parquet

Read and query a parquet file with optional filters and enhanced operators.

add_record

Add a new record to a parquet file and create an audit log entry.

update_records

Update existing records matching filters and create an audit log entry.

upsert_record

Insert or update a record with enhanced filters to prevent duplicates; returns action and audit IDs.

delete_records

Delete records matching filters and create an audit log entry.

get_statistics

Get basic statistics about a parquet file.

read_audit_log

Read audit log entries with optional filters to view modification history.

rollback_operation

Rollback a specific operation using its audit ID and create an inverse operation.

search_parquet

Semantic search using embeddings to find semantically similar records.

generate_embeddings

Generate embeddings for text fields to enable semantic search.