bitsandbytes_skill

Overview

This skill quantizes large language models to 8-bit or 4-bit formats to reduce GPU memory by roughly 50–75% with minimal accuracy loss. It supports INT8, NF4, FP4 formats, QLoRA fine-tuning, and 8-bit optimizers, and integrates with HuggingFace Transformers for load, inference, and training. Use it to fit larger models on constrained GPUs, speed up inference, or enable memory-efficient training workflows.

How this skill works

The skill configures BitsAndBytes quantization options and loads models via HuggingFace Transformers with device_map and quantization_config. It offers 8-bit (load_in_8bit) for better accuracy and 4-bit (load_in_4bit with nf4/double quant) for maximum memory savings, plus 8-bit optimizers and QLoRA steps for low-memory fine-tuning. It also provides tips for CPU offload, memory estimates, and monitoring during load or training.

When to use it

• You need to run a larger LLM on limited GPU VRAM.
• You want memory-efficient inference with minimal accuracy loss.
• You want to fine-tune large models on a single/consumer GPU using QLoRA.
• You want to reduce optimizer memory with 8-bit Adam/AdamW during training.
• You need faster prototype iteration and lower-cost GPU usage.

Best practices

• Estimate memory needs before choosing quantization (FP16/INT8/INT4 formulas).
• Prefer 8-bit for accuracy-critical inference and 4-bit (NF4 + double quant) for max memory savings.
• Use device_map='auto' and CPU/disk offload when loading very large models to avoid OOMs.
• When fine-tuning, run prepare_model_for_kbit_training and add LoRA adapters to keep trainable params small.
• Monitor CUDA memory and test a short generation to validate behavior after loading.

Example use cases

• Load Llama-2-7B in 8-bit to halve VRAM usage for inference on a 16GB GPU.
• Fine-tune a 7B model with QLoRA on a single 24GB GPU and save tiny LoRA adapters.
• Train with an 8-bit AdamW optimizer to cut optimizer state memory by ~75%.
• Run a 13B model in 4-bit on a 24GB GPU for low-cost experimentation and research.
• Offload parts of a 70B model to CPU/disk and use quantization to enable multi-GPU or single-GPU workflows.