MediaCrawler

MediaCrawler MCP Service lets you access social media data through a standardized MCP interface, enabling AI assistants to directly call and analyze high-quality content with a stable, self-contained workflow.

How to use

You connect your MCP client to the server using the MCP SSE endpoint and use registered tools to fetch and analyze social media data. Start by launching the server, open the management UI to login and configure platforms, then call tools such as search, detail, and comments to gather structured data for AI analysis. Tools are grouped by platform and can be invoked directly from your MCP client or via the online inspector for testing.

When you operate from an AI assistant, point the MCP client at the server’s SSE URL, pick the appropriate tool name (for example bili_search or xhs_search), and supply the required parameters. Use the inspector to verify available tools and their arguments before integrating them into your prompts. The output is designed to be flat JSON without redundant nesting, making it easier for AI models to consume.

How to install

Prerequisites: Python 3.13+, Redis, Chrome/Chromium, and an optional Node.js 16+ if you plan to use additional tooling.

# 1) Clone the project
git clone <your-repo-url>
cd media-crawler-mcp-service

# 2) Install dependencies
poetry install
poetry run playwright install chromium

# 3) Configure environment
cp .env.example .env
# Edit .env to adjust ports, platform switches, Redis, etc.

# 4) Start the service
redis-server
poetry run python main.py

Management and debugging UIs are available at the following addresses:

• Management: http://localhost:9090/admin
• Inspector (tool tester): http://localhost:9090/admin/inspector

Configuration and security notes

The server supports externalized login via a visual interface and QR/Cookie-based modes, with persistent login states to improve stability of data collection. For public deployment, consider adding a gateway to enforce authentication. Local and internal network deployments do not require authentication by default.

Data persistence and routing are organized by platform codes (e.g., bili for Bilibili, xhs for Xiaohongshu). All data outputs are flattened for AI-friendly consumption, avoiding deep nesting and noise.

Using from an AI assistant (examples)

Example tool usage from an AI assistant typically involves calling a tool with a structured payload. Use the inspector to verify available tools and expected arguments for the current server state.

Tools overview and platform grouping

The MCP tooling is organized by platform groups such as bili and xhs. Each tool corresponds to a specific data retrieval action like search, detail, creator, and comments. Tools are designed to be reusable and easy to integrate into automated workflows.

Notes on architecture and workflow

The system emphasizes layered decoupling, verified schemas via models, cached state, and rate-friendly interactions. Browser contexts are reused to optimize performance and minimize anti-bot friction. Output data is structured for AI consumption with minimal noise.

Troubleshooting and common scenarios

If startup fails, ensure Redis is running, dependencies are installed, and the server starts with the expected command. If the login state is lost, verify Redis stability and network connectivity to avoid frequent re-authentication. For empty or slow searches, prefer bili_search first, reduce page_size, and increase inter-request delays.

Available tools

service_info

Provide service information to verify the MCP server is up and its basic configuration.

service_health

Health check endpoint to confirm the MCP server is operating normally.

list_tools

Return a list of all registered MCP tools available for the current session.

tool_info

Retrieve detailed information about a specific MCP tool, including required arguments.

bili_search

Search Bilibili content using keywords and pagination, returning structured results suitable for AI consumption.

bili_crawler_detail

Fetch detailed information for a specific Bilibili video by ID(s).

bili_crawler_creator

Retrieve content and profile information for a Bilibili creator.

bili_search_time_range_http

Search Bilibili content within a specified time range via HTTP-based query.

bili_crawler_comments

Collect comments for a Bilibili video or post to analyze engagement.

xhs_search

Search Xiaohongshu notes with keywords and pagination.

xhs_crawler_detail

Fetch detailed Xiaohongshu note information (requires xsec_token).

xhs_crawler_creator

Retrieve Xiaohongshu creator作品 and profile data.

xhs_crawler_comments

Collect comments on Xiaohongshu notes for sentiment and engagement insights.