Welcome to veRL/HybridFlow’s documentation!

veRL (HybridFlow) is a flexible, efficient and industrial-level RL(HF) training framework designed for large language models (LLMs) Post-Training.

veRL is flexible and easy to use with:

• Easy to support diverse RL(HF) algorithms: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.

• Seamless integration of existing LLM infra with modular API design: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.

• Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

• Readily integration with popular Hugging Face models

veRL is fast with:

• State-of-the-art throughput: By seamlessly integrating existing SOTA LLM training and inference frameworks, veRL achieves high generation and training throughput.

• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

---

Preparation

• 1. Installation
• 2. Prepare Data (Parquet) for Post-Training
• 3. Implment Reward Function for Dataset

PPO Example

• 1. PPO Example Architecture
• 2. Config Explaination
• 3. GSM8K Example

PPO Trainer and Workers

• PPO Ray Trainer
• PyTorch FSDP Backend
• Megatron-LM Backend

Advance Usage and Extension

• Ray API Design Tutorial
• Extend to other RL(HF) algorithms
• Add models to FSDP backend
• Add models to Megatron-LM backend

Contribution

veRL is free software; you can redistribute it and/or modify it under the terms of the Apache License 2.0. We welcome contributions. Join us on GitHub .