llama-cpp_skill

Overview

This skill runs LLM inference on CPU, Apple Silicon, and consumer GPUs without requiring NVIDIA CUDA. It supports GGUF quantized models (1.5–8 bit) to reduce memory and greatly speed up inference on edge and desktop hardware. Use it to deploy performant models on M1/M2/M3 Macs, AMD/Intel GPUs, Raspberry Pi, or any environment where CUDA is unavailable.

How this skill works

The skill builds a lightweight C/C++ inference engine that loads GGUF-format models and executes optimized kernels for CPUs, Metal (Apple), ROCm (AMD), and optional CUDA when available. It supports layer offloading, quantization-aware I/O, and server mode with an OpenAI-compatible HTTP endpoint. Quantized formats (Q2–Q8 variants) trade memory and runtime speed for output fidelity, allowing models that normally require large GPUs to run on commodity hardware.

When to use it

• Deploy on machines without NVIDIA GPUs (CPU-only, Apple Silicon, AMD/Intel GPUs).
• Run models on edge devices like Raspberry Pi or small servers where minimal dependencies are required.
• Speed up CPU inference with GGUF quantization to fit large models into limited memory.
• Provide an OpenAI-compatible local server for privacy-sensitive or offline environments.
• Avoid heavy Python or Docker runtimes and prefer simple C/C++ builds.

Best practices

• Choose Q4_K_M as the default balance of speed and quality; use Q6_K or Q8_0 for highest fidelity when memory allows.
• Build with Metal on Apple Silicon (LLAMA_METAL=1) to leverage GPU acceleration and reduce latency.
• Offload layers to GPU (ngl flag) for hybrid CPU+GPU setups to fit large models into device memory.
• Test multiple quantization levels on representative prompts to validate quality vs. throughput trade-offs.
• Increase context size only if the model supports it and monitor memory usage for large windows.

Example use cases

• Local chat assistant on an M3 Mac using a 7B GGUF-quantized model for privacy-preserving interactions.
• Edge analytics on Raspberry Pi with a small quantized model for on-device inference and offline operation.
• Batch text generation on CPU-only servers where containerized Python is not required.
• Hybrid deployments that offload top layers to a consumer GPU while keeping most computation on CPU to fit 70B models.
• Run a private OpenAI-compatible text generation API on-premise for latency-sensitive applications.