faiss_skill

Overview

This skill packages FAISS for high-performance similarity search and clustering of dense vectors. It exposes index types (Flat, IVF, HNSW, PQ), GPU acceleration, and tools for billion-scale k-NN retrieval. Use it when throughput, latency, and memory efficiency matter for pure vector search workloads.

How this skill works

The skill builds and queries vector indices optimized for nearest-neighbor search. It supports exact (Flat) and approximate (IVF, HNSW, PQ) indexes and can move CPU indices to GPU for large speedups. It also provides training, serialization, and integrations with common vector-store wrappers for retrieval pipelines.

When to use it

• You need fast k-NN search on millions to billions of dense vectors.
• GPU acceleration is required to meet latency or throughput targets.
• You only need pure vector similarity without metadata filtering.
• Batch or offline embedding processing with large-scale indexes.
• Memory-constrained deployment where quantization (PQ) helps.

Best practices

• Pick index type by scale: Flat for small datasets, IVF for medium, HNSW for high quality, PQ for memory efficiency.
• Normalize vectors and use inner-product indexes for cosine similarity.
• Train and save costly indexes once; reuse serialized indices to avoid retraining.
• Tune search knobs (nprobe for IVF, ef_search for HNSW) to balance accuracy and speed.
• Use GPU indices for large datasets and batch queries to maximize throughput.

Example use cases

• Semantic search over millions of document embeddings for low-latency retrieval.
• Nearest-neighbor lookup for recommendation engines at large scale.
• Embedding-based deduplication and clustering of image or text representations.
• Offline vector index building with periodic updates and fast serialized loads.
• High-throughput similarity search in pipelines requiring GPU-backed acceleration.