verl_skill

Overview

This skill provides practical guidance for training large language models with reinforcement learning using verl (Volcano Engine RL). It focuses on production-ready workflows, configuration recipes, and troubleshooting for algorithms like PPO, GRPO, and others across flexible backends. Use it to scale RLHF and post-training optimization for models from small to multi-hundred-billion parameters.

How this skill works

The guidance explains the HybridFlow architecture verl uses: a single-process controller (Ray) that orchestrates multi-process workers for rollout, reward, and training. It covers selecting rollout engines (vLLM, SGLang, HF), choosing training backends (FSDP, Megatron), and configuring algorithms (GRPO, PPO/GAE, REINFORCE++). Examples include dataset prep, reward function hooks, config templates, and launch commands for single- and multi-node runs.

When to use it

• When you need production-ready RL training at scale (tested to 671B parameters).
• When you require flexible backend swaps: FSDP, Megatron-LM, vLLM, or SGLang.
• When implementing RLHF, GRPO, PPO/GAE, RLOO, or custom advantage estimators.
• When training vision-language or multi-turn agentic models with tool calling.
• When you want LoRA-in-RL, sequence/expert parallelism, or multi-turn rollouts.

Best practices

• Start with a small-scale repro on 1–2 nodes to validate reward and stability before scaling.
• Use parquet datasets with prompt and reward fields; validate reward function offline.
• Enable KL control and tune kl_loss_coef to avoid policy collapse during early training.
• Monitor WandB/TensorBoard for reward trends and loss spikes; reduce LR or increase KL if unstable.
• Match vLLM and backend versions; prefer Docker for reproducible production deployments.

Example use cases

• Train a math-reasoning model (GSM8K) with GRPO and rule-based reward extraction.
• Run PPO with a separate critic (GAE) for dense-reward tasks like summarization quality.
• Scale >70B training using Megatron bridge and multi-node NVLink clusters for expert parallelism.
• Fine-tune a vision-language model with vLLM rollouts and enable_vision in rollout config.
• Apply LoRA-in-RL to reduce GPU memory and speed up iteration while keeping policy updates.