vllm_skill

Overview

This skill serves large language models with high throughput and low latency using vLLM's PagedAttention and continuous batching. It provides OpenAI-compatible endpoints, supports quantization (GPTQ/AWQ/FP8), and can run models across tensor-parallel GPUs. Use it to deploy production APIs or run large offline batches while minimizing GPU memory usage.

How this skill works

The skill launches vLLM to host models with a block-based KV cache (PagedAttention) and continuous batching that mixes prefill and decode phases for maximum throughput. It exposes an OpenAI-compatible REST interface, Prometheus metrics, and flags for tensor parallelism and quantization. vLLM handles automatic batching internally and supports prefix caching, speculative decoding, and GPU memory tuning.

When to use it

• Deploy production LLM APIs requiring 100+ req/sec and low TTFT
• Serve large models on limited GPU RAM using quantization
• Provide OpenAI-compatible endpoints for multi-user apps or chatbots
• Run high-throughput offline batch inference across many prompts
• Optimize inference throughput with tensor parallelism and continuous batching

Best practices

• Start with a staged rollout: configure, test with limited traffic, enable monitoring, then deploy
• Tune gpu-memory-utilization and max-model-len to avoid OOM during load
• Use AWQ/GPTQ/FP8 quantization to fit larger models into available VRAM
• Enable prefix caching and chunked prefill for repeated or long prompts to reduce TTFT
• Monitor vllm Prometheus metrics (TTFT, running requests, GPU cache usage) and target >80% GPU utilization

Example use cases

• Production chat API serving Llama-family models with OpenAI-compatible endpoints
• Batch processing thousands of prompts offline with automatic internal batching
• Serving a 70B model on constrained hardware using AWQ quantization and tensor parallelism
• Benchmarking and optimizing TTFT and throughput for a multi-tenant assistant
• Deploying a Dockerized vLLM service behind a load balancer with Prometheus metrics