Crawl4AI RAG

Provides a production-ready RAG solution combining Crawl4AI content extraction, vector search, RAM mode, and MCP-based AI assistant integration.

python

0

GitHub Stars

python

Language

5 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

You run a high-performance Retrieval-Augmented Generation (RAG) MCP server that uses Crawl4AI for web content extraction, sqlite-vec for vector storage, RAM database mode for fast queries, and MCP integration for AI assistants. This setup enables you to crawl and index content, perform fast semantic searches, and interact with AI clients via MCP for seamless question-answering and content discovery.

How to use

Use an MCP client to connect to the server and perform common workflows such as crawling new content, storing it, searching semantically, and asking an AI assistant to reason over your indexed material. You can crawl web pages with intelligent extraction, clean and normalize content, then query the in-memory vector store for fast retrieval. Typical usage patterns include adding new URLs, running semantic searches with tags, and asking the assistant to expand on relevant topics.

How to install

Prerequisites you need before installing and running the MCP server:

Python 3.11+ and a virtual environment tool (venv) installed on your system
Docker and Docker Compose if you plan to run the full server deployment

Follow these concrete steps to set up locally and run the MCP server

Clone the project repository

git clone https://github.com/Rob-P-Smith/mcpragcrawl4ai.git
cd mcpragcrawl4ai

Create and activate a Python virtual environment, then install dependencies

python3 -m venv .venv
source .venv/bin/activate  # Linux/Mac
pip install -r requirements.txt

Start the Crawl4AI service in a container

docker run -d --name crawl4ai -p 11235:11235 unclecode/crawl4ai:latest

Create a local environment configuration file with required variables

# Create .env file
cat > .env << EOF
IS_SERVER=true
USE_MEMORY_DB=true
LOCAL_API_KEY=dev-api-key
CRAWL4AI_URL=http://localhost:11235
EOF

Run the MCP server

python3 core/rag_processor.py

Additional deployment options

If you prefer a Docker-based deployment for production, you can run the full server with REST API and MCP endpoints using Docker Compose.

Deploy the full server stack

cd mcpragcrawl4ai
docker compose -f deployments/server/docker-compose.yml up -d

Verify the deployment is healthy

curl http://localhost:8080/health

Available tools

crawl_and_remember

Crawl a URL, extract content with Crawl4AI, convert to markdown, and store the content in the RAG database with associated tags.

search_memory

Perform a semantic search over stored content using vector embeddings to find relevant pages or passages.

target_search

Execute an intelligent search with initial results and automatic tag expansion to improve recall.

get_database_stats

Retrieve statistics about the current database, including pages, embeddings, and memory usage.