grepai-chunking_skill

This skill helps optimize code chunking for embeddings by tuning size and overlap to balance precision and context.

1

GitHub Stars

1

Bundled Files

2 months ago

Catalog Refreshed

4 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill yoanbernabeu/grepai-skills --skill grepai-chunking

SKILL.md7.5 KB

Overview

This skill configures how GrepAI splits source files into chunks for embedding to optimize semantic code search. It helps you balance chunk size and overlap to improve search precision, preserve context, and control index size. Use it to tune indexing behavior for different languages and codebase shapes.

How this skill works

The skill sets target token size and overlap for chunking: files are divided into segments, each segment gets its own embedding, and adjacent segments can share tokens to preserve boundary context. GrepAI favors logical split points (empty lines, closing braces, statement ends), so actual chunk boundaries may vary slightly from targets. You can re-index after changing settings to compare the effect on search results and index size.

When to use it

When search results are too fragmented or missing context
When search returns too much irrelevant code (too much noise)
When tuning index size and storage vs. search quality
When adapting chunking for verbose or concise programming languages
When diagnosing unexpected search behavior after indexing

Best practices

Start with default size 512 and overlap 50; it balances precision and context
Increase size for verbose languages or large classes (e.g., 768) to capture full methods
Decrease size for concise languages or microservices (e.g., 384) to improve precision
Adjust overlap to preserve function boundaries; use 50 as standard, 100 for more context
Always re-index after changing settings and test with representative queries

Example use cases

Microservices repo with many small functions: set size=384 overlap=40 to capture single functions
Monolithic Java codebase: set size=768 overlap=75 to keep methods intact and reduce fragmentation
Mixed-language repo: keep balanced default (512/50) and tweak per-language overrides where needed
Diagnosing search drift: run before/after searches and diff results after re-indexing
Reducing index storage: increase chunk size or reduce overlap to lower total chunks

FAQ

GrepAI uses approximate token counting (about 4 characters per token for English-style text); code token counts vary with identifiers and syntax.

What if chunk boundaries cut a function?

Increase overlap or chunk size so logical boundaries like closing braces and empty lines are included in adjacent chunks.

How does overlap affect index size?

Higher overlap increases the number of chunks and therefore index size; typical overlap (50) adds a small overhead, while 100 can add ~10% more chunks.