gguf_skill

Overview

This skill packages GGUF conversion and llama.cpp quantization workflows to enable efficient CPU and Apple Silicon inference with flexible 2–8 bit quantization. It guides conversion from HuggingFace formats, generation of importance matrices, and producing K-quant or legacy quantized GGUF files for local deployment. The materials focus on practical commands, Python examples, and hardware-specific optimizations for consumer and server setups.

How this skill works

The skill explains how to convert pretrained models to the GGUF file format and apply quantization methods (Q2_K to Q8_0 and legacy types) using llama.cpp tools. It covers optional importance-matrix calibration (imatrix) to improve low-bit quality, building llama.cpp for CPU, Metal, or CUDA, and running inference via CLI, Python bindings, or an OpenAI-compatible server. It also shows workflows for producing multiple quantizations from a single base model to trade off quality and footprint.

When to use it

• Deploying models on laptops, desktops, or Apple Silicon without a full GPU stack
• Reducing memory and storage footprint for local inference with 2–8 bit quantization
• Creating multiple quantized variants (Q4_K_M, Q5_K_M, Q6_K, Q8_0) to compare quality/size
• Needing pure C/C++ inference or integration with local tools (LM Studio, Ollama, text-generation-webui)
• Running experimental or production local servers compatible with OpenAI APIs

Best practices

• Prefer K-quant methods (Q4_K_M or Q5_K_M) for the best quality/size balance
• Generate an importance matrix (imatrix) for Q4 and lower to improve output quality
• Offload as many layers as VRAM allows for GPU acceleration; use n_gpu_layers for Metal/CUDA
• Match CPU threads to physical cores and tune n_batch for prompt throughput
• Start with a 4096 context window and increase only if needed to avoid memory spikes

Example use cases

• Convert a HuggingFace FP16 model to GGUF, quantize to Q4_K_M, and run local inference on Apple Silicon
• Create imatrix calibration from diverse text, generate multiple quantized builds, and benchmark latency/quality
• Run a local OpenAI-compatible server for on-premise chat with a quantized GGUF model
• Integrate a small-footprint GGUF model into text-generation-webui or LM Studio for offline use
• Batch-quantize a set of models with a single imatrix to produce Q4_K_M, Q5_K_M, and Q6_K variants