Dev Tool

You run an MCP Server that provides web crawling, browser automation, and multi-format content extraction. It lets you crawl pages, inspect content and network activity, and save outputs in HTML, JSON, PDF, Markdown, and images, all under a configurable MCP interface.

How to use

Use an MCP client to connect to the server and invoke tools that perform crawling, content extraction, and page analysis. You can crawl a web page, retrieve its complete content, capture console messages, and monitor network requests. The server streams progress during long crawls and saves outputs in multiple formats for easy consumption.

How to install

Prerequisites: Install Python 3.8 or higher and the pip package manager. You also need Playwright browsers for headless browser automation.

Step by step commands to set up and run the MCP server on your machine:

# 1) Clone the repository
# (Replace with your actual repo path if you are cloning manually)
# git clone <repository-url>
# cd mcp-server

# 2) Install the package and dependencies
pip install -e .

# 3) Install Playwright browsers (Chromium)
python -m playwright install chromium

Configuration

Environment variable for testing is supported: TEST_URL. You can set it in your shell or your deployment environment. The MCP server exposes a set of tools that you can call through the MCP protocol, including simple helpers and web crawling capabilities.

Security

The server validates URLs, enforces allowed protocols (http/https), and sanitizes inputs to prevent injection attacks. Browser automation runs in headless mode with restricted privileges, and saved files are written only to explicitly chosen paths to minimize risk.

Troubleshooting

If you encounter browser-related issues, ensure Playwright browsers are installed with the correct command. For permission errors, verify write permissions on the save paths. For network issues, confirm the target URL is accessible and that there are no firewall blocks. For large crawls, monitor memory usage and clean up temporary files as needed.

Testing

Tests cover both crawler and browser functionalities. Run the test suite to verify HTML/JSON/PDF/Markdown outputs, file saving, and browser interactions.

Deployment

In production, install the MCP package, ensure Chromium is available for Playwright, and start the MCP server so clients can connect via the MCP protocol.

Notes

The server supports multiple ways to start in your environment. For local development, you can launch via a script in your project or run the server module directly.

Example: MCP configuration snippets

{
  "mcpServers": {
    "dev_tool_mcp": {
      "command": "/bin/bash",
      "args": [
        "-c",
        "cd /absolute/path/to/your/dev-tool-mcp && ./launch_server.sh"
      ],
      "description": "MCP development tool server providing web crawling, browser automation, content extraction, and real-time page analysis capabilities"
    },
    "dev_tool_mcp_shell": {
      "command": "/absolute/path/to/your/dev-tool-mcp/launch_server.sh",
      "args": [],
      "description": "MCP development tool server providing web crawling, browser automation, content extraction, and real-time page analysis capabilities"
    },
    "dev_tool_mcp_console": {
      "command": "/absolute/path/to/your/dev-tool-mcp/launch_server.sh",
      "args": [],
      "description": "MCP development tool server providing web crawling, browser automation, content extraction, and real-time page analysis capabilities"
    },
    "dev_tool_mcp_win": {
      "command": "cmd",
      "args": [
        "/c",
        "cd /d C:\\absolute\\path\\to\\your\\dev-tool-mcp && launch_server.bat"
      ],
      "description": "MCP development tool server providing web crawling, browser automation, content extraction, and real-time page analysis capabilities"
    }
  }
}

Available tools

say_hello

A simple tool that returns a greeting message. You can pass an optional name to personalize the greeting.

echo_message

Echoes back the provided message exactly as submitted.

crawl_web_page

Crawls a web page and saves output in HTML, JSON, PDF, and Markdown formats, downloads linked resources, and can capture a screenshot if requested.

get_page_content

Fetches the complete content of a page including HTML structure, metadata, and links.

get_console_messages

Captures console messages emitted by the page, including logs, warnings, and errors.

get_network_requests

Monitors and reports network requests made by the page, includingURLs, status, headers, and timing.