llm-integration_skill

Overview

This skill provides expert guidance and reusable patterns for integrating local Large Language Models using llama.cpp and Ollama. It focuses on secure model loading, inference optimization, prompt handling, and defenses against LLM-specific attacks like prompt injection, model theft, and DoS. The content targets reliable, low-latency, privacy-preserving local deployments for assistants and services.

How this skill works

The skill inspects recommended configurations, code patterns, and runtime controls for local LLMs. It describes secure clients for Ollama and llama-cpp-python, prompt sanitization, resource-limited inference, streaming with output filtering, checksum verification, and token optimization. Implementation is framed with TDD, example tests, and stepwise hardening to keep behavior deterministic and safe.

When to use it

• Deploying local LLM inference for voice assistants or on-device agents where privacy is required
• Building secure API endpoints that forward user prompts to local models
• Optimizing latency and memory for real-time conversational systems
• Implementing defenses against prompt injection, data exfiltration, or resource exhaustion
• Orchestrating multiple models with clear security boundaries and fallbacks

Best practices

• Always validate and sanitize incoming prompts; detect common injection patterns before inference
• Run model inference in isolated/sandboxed processes with RLIMITs and timeouts to prevent DoS
• Verify model files with checksums and restrict model paths to an allowlisted directory
• Use quantization and appropriate thread settings based on hardware; prefer streaming for low TTFB
• Mock LLM responses in tests (TDD) so security and behavior are deterministic before deployment

Example use cases

• JARVIS-style on-device voice assistant with sub-second streaming responses and filtered output
• Privacy-first customer support agent that never sends prompts to the cloud and logs metrics safely
• Multi-model router that selects a small fast model for short tasks and a larger model for complex reasoning
• Secure inference endpoint that truncates, sanitizes, and rate-limits prompts to prevent abuse
• Edge deployment using 4-bit/8-bit quantized models to reduce memory while preserving throughput