gptq_skill

Overview

This skill provides post-training 4-bit quantization for large language models (LLMs) using group-wise GPTQ to preserve accuracy while drastically reducing memory and improving inference speed. It targets deployment of models from 13B up to 405B on consumer and datacenter GPUs, integrates with transformers and PEFT, and supports multiple kernel backends for different hardware.

How this skill works

GPTQ divides weight matrices into small groups (commonly 128 elements) and computes per-group quantization scales and zero-points, using second-order (Hessian) information to minimize error. The result is 4-bit weights with near-FP16 accuracy. The skill includes tools to load pre-quantized models, quantize models with calibration data, save/push quantized artifacts, and choose optimized kernel backends (ExLlama, Marlin, Triton) for your GPU.

When to use it

• Deploy 70B+ models on limited GPU memory (single A100, consumer RTX 4090) to avoid OOM.
• Need ~4× model size reduction with <2% perplexity degradation versus FP16.
• Want 3–5× faster inference than FP16 on supported GPUs.
• Combine with QLoRA/PEFT for memory-efficient fine-tuning of k-bit models.
• Prepare production inference stacks with multi-GPU or CPU offload.

Best practices

• Start with group_size=128 (recommended default) for a balance of accuracy and speed.
• Use representative calibration data (100–200 short sequences) when quantizing from FP16.
• Prefer ExLlama on consumer GPUs and Marlin on Ampere+ hardware for best throughput.
• Enable device_map and max_memory for automatic multi-GPU splits; use manual mapping for tight control.
• Test perplexity and sample outputs after quantization to validate quality before production.

Example use cases

• Run Llama-2-13B or 70B chat models on a single 80GB GPU with 4× memory savings.
• Deploy 13B models on an RTX 4090 for low-cost inference with competitive latency.
• Fine-tune a GPTQ-quantized model with LoRA adapters (QLoRA) to enable large-model training on limited hardware.
• Batch inference for multi-prompt pipelines using quantized weights to increase throughput.
• Mix GPU and CPU offload to host 405B models across a multi-node system.