model-quantization_skill

Overview

This skill is an expert toolkit for AI model quantization and deployment optimization. It focuses on 4-bit and 8-bit quantization, GGUF conversion for llama.cpp, memory footprint reduction, and clear quality-versus-performance tradeoffs for resource-constrained JARVIS environments. Use it to produce validated, deployable model variants that balance latency, memory, and accuracy.

How this skill works

The skill provides a secure quantization pipeline that converts models to GGUF, runs quantize binaries, and verifies checksums. It includes test-driven workflows: failing tests first, minimal implementation to pass, refactoring, and full verification with benchmarks for perplexity, latency, and memory. Selector logic recommends quantization levels based on model size, available RAM, and quality priority.

When to use it

• Deploy LLMs to consumer hardware with limited RAM/CPU
• Prepare models for on-device voice assistants with strict latency budgets
• Create multiple model tiers (low-RAM, balanced, high-quality) for heterogeneous hardware
• Reduce memory footprint to fit models into edge servers or small VMs
• Validate that quantized models meet quality and security requirements

Best practices

• Start with test-driven quantization: write quality and memory tests before converting models
• Validate model integrity with SHA256 checksums before and after conversion
• Use calibration and layer-wise quantization for sensitive layers to preserve quality
• Benchmark perplexity, latency, and memory on representative prompts and workloads
• Select quantization level based on usable RAM (reserve ~2GB overhead) and quality priority

Example use cases

• Convert a HuggingFace model to GGUF and produce Q5_K_M and Q4_K_M variants for low- and mid-tier devices
• Automate CI tests that assert quantized model perplexity stays within 10% of baseline and accuracy above 90% for critical prompts
• Deploy a 7B model in an 8GB RAM environment by selecting Q5_K_M and verifying memory/latency budgets
• Build a conversion pipeline that optionally runs quantize and stores .sha256 checks alongside outputs
• Run nightly benchmarks to detect regressions in latency or quality after model changes