openrlhf_skill

Overview

This skill packages a high-performance RLHF training framework optimized for large LLMs (7B–70B+). It leverages Ray for distributed orchestration, vLLM for accelerated inference, and ZeRO-3 for memory efficiency to deliver fast, scalable PPO/GRPO/RLOO/DPO workflows. Designed for multi-node GPU clusters, it focuses on throughput and practical training stability improvements.

How this skill works

The framework launches distributed actors for reference, reward, critic, and actor models using Ray and shares GPU resources via a Hybrid Engine that integrates vLLM and DeepSpeed-style zero optimization. It supports multiple RLHF algorithms (PPO, GRPO, RLOO, DPO) and provides vLLM engines for fast generation during rollouts. Configuration flags control colocation, GPU utilization, KL handling, and batching to tune performance and memory usage.

When to use it

• Training large models (7B–70B+) with reinforcement learning from human feedback
• Running distributed multi-node RLHF workloads with Ray and GPU clusters
• Need fast inference acceleration for rollouts using vLLM
• Comparing or running PPO, GRPO, RLOO, or DPO within a single framework
• Optimizing GPU utilization and reducing idle time via Hybrid Engine GPU sharing

Best practices

• Run on NVIDIA A100/H100 class GPUs and use bf16 where supported to save memory
• Use Hybrid Engine and vLLM sleep modes to minimize GPU idle time and improve throughput
• Start with smaller batch sizes and enable gradient_checkpointing and ZeRO-3 for memory-limited setups
• Tune init_kl_coef and monitor KL drift; increase KL if training becomes unstable
• When encountering OOMs, disable model colocation and allocate dedicated GPUs per model role

Example use cases

• Full RLHF pipeline: SFT → Reward Model (DPO) → PPO with vLLM-accelerated rollouts
• Memory-efficient GRPO training to avoid a separate critic and reduce GPU use
• DPO preference optimization as a lightweight alternative when no reward model is desired
• Benchmarking algorithm trade-offs (PPO vs GRPO vs RLOO vs REINFORCE++) on multi-node clusters
• Fast prototype runs using Ray job submit and Hybrid Engine flags to validate configs before scale-up