cocoindex_skill

Overview

This skill provides a comprehensive toolkit for developing with the CocoIndex library to build real-time, incremental ETL flows for AI data processing. It helps you define indexing pipelines in Python, create reusable custom functions, and operate flows via CLI or API. Use it to produce vector indexes, knowledge graphs, or live syncs from diverse data sources to multiple targets.

How this skill works

The skill guides you through designing flows that import sources, apply transformations (chunking, embedding, LLM extraction, custom ops), collect outputs, and export to targets like Postgres+pgvector, Qdrant, LanceDB, Neo4j, or Kuzu. It enforces flow patterns (use .row(), assign transformed values to fields, define collector exports) and supports incremental processing and live updates. It also covers dependency installation, environment setup (database and LLM keys), function typing, and running/testing flows via the CLI or programmatic API.

When to use it

• You need to build a vector search index for documents, code, or images.
• You want an embedding pipeline with local or cloud models and incremental updates.
• You must extract structured data with LLMs into a knowledge graph or relational store.
• You need to write custom transformation functions with strict return type annotations.
• You want live syncing of source changes to targets or periodic incremental processing.

Best practices

• Always ask for source details (location, file types, change frequency) before designing a flow.
• Assign transformation results to flow fields, not local variables, so CocoIndex can infer schema and track changes.
• Annotate custom function return types precisely (vectors with fixed dimension, structs, tables) to enable correct target schemas and vector indexes.
• Select CocoIndex extras based on needs (embeddings, colpali, lancedb) and verify environment variables for COCOINDEX_DATABASE_URL and required LLM API keys.
• Test flows locally using cocoindex update --setup and run cocoindex server to validate ingestion and view CocoInsight.

Example use cases

• Index a corpus of PDFs into a Qdrant vector store with 768-dim embeddings and a searchable metadata schema.
• Stream S3 image uploads, generate captions via a vision API, embed captions, and export to LanceDB for multimodal search.
• Extract named entities and relations from documents with an LLM and export results to Neo4j to build a knowledge graph.
• Create a one-time migration flow that reads Postgres rows, transforms text, and writes vectors to Postgres+pgvector for semantic search.