NVIDIA AI Unveils ProRL Agent: A Decoupled Rollout-as-a-Service Infrastructure for Reinforcement Learning of Multi-Turn LLM Agents at Scale

NVIDIA researchers have recently introduced ProRL AGENT, a scalable infrastructure specifically designed for the reinforcement learning (RL) training of multi-turn LLM agents. This innovative system adopts a ‘Rollout-as-a-Service’ approach, separating agentic rollout orchestration from the training loop. This shift in architecture addresses the resource conflicts between I/O-intensive environment interactions and GPU-intensive policy updates that currently hinder agent development.

The Core Problem: Tight Coupling

Tasks involving multi-turn agents require interaction with external environments, such as code repositories or operating systems, through iterative tool usage. Many existing frameworks embed rollout control directly within the training process, leading to two primary limitations:

1. Conflicting System Requirements: Rollouts are I/O-bound, necessitating sandbox creation, long-lived tool sessions, and asynchronous coordination. Meanwhile, training is GPU-intensive, focusing on forward/backward passes and gradient synchronization. Running both processes in one system causes interference and reduces hardware efficiency.
2. Maintenance Barriers: Embedding rollout logic in the trainer makes it challenging to transition to different training backends or support new runtime environments without re-implementing the execution pipeline.
   

   System Design: Rollout-as-a-Service
   

   ProRL AGENT functions as a standalone HTTP service responsible for managing the complete rollout lifecycle. The RL trainer interacts with the server exclusively through an API, remaining unaware of the underlying rollout infrastructure.

   Three-Stage Asynchronous Pipeline
   

   To maximize throughput, the server coordinates rollouts through an asynchronous three-stage process:

3. INIT: Initialization workers set up sandbox containers and configure tools.
4. RUN: Rollout workers drive the multi-turn agent loop and gather trajectories.
5. EVAL: Evaluation workers assess results against ground truth to generate reward signals.

   By assigning each stage to an independent worker pool, ProRL AGENT allows phases to overlap across different jobs, preventing slow evaluations from impeding the rollout process.

   HPC-Compatible Sandboxing and Optimized Tools
   

   ProRL AGENT leverages Singularity for its sandbox infrastructure, enabling rootless execution necessary for deployment on shared HPC clusters managed by Slurm. The system also incorporates several optimizations to reduce tool execution latency, a significant component of total rollout time:

   • Efficient Bash: Replaces tmux-based terminal multiplexing with a ptyprocess-based direct pseudo-terminal, reducing shell command latency.
   • Direct IPython API: Establishes connections to persistent kernels via an in-process API, eliminating networking overhead.
   • Unix Domain Sockets (UDS): Substitutes TCP loopback for communication between the agent and the execution server inside the container, minimizing additional latency.
     

     Advanced Features for Scalable RL
     

     ProRL AGENT introduces various mechanisms to enhance training stability and hardware utilization:

     Load Balancing and Prefix Cache Reuse
     

     The server manages a pool of LLM inference backends using a min-heap keyed by assignment counts. Once a task is assigned, all subsequent calls within that task are directed to the same backend, maximizing prefix cache reuse and reducing inference time across multiple agent turns.

     Token-in/Token-out Communication
     

     To eliminate re-tokenization drift, where the token sequence generated during rollout differs from that used during training, ProRL AGENT employs token IDs as the canonical representation throughout the entire process. Log-probabilities and IDs are propagated unchanged from the inference backend to the trainer.

     Optimized DAPO Implementation
     

     The system supports Dynamic Sampling Policy Optimization (DAPO), filtering out ‘non-informative’ prompts that yield uniform rewards. ProRL AGENT incorporates an asynchronous replenishment mechanism to maintain maximum throughput, terminating redundant active jobs early once the target number of informative prompts is reached.

     Experimental Results on SWE-Bench Verified
     

     The system underwent validation using Qwen3 models across various scales, consistently outperforming reproduced baselines. ProRL AGENT demonstrated versatility in STEM, Math, and Code domains, displaying steady reward growth during RL training. Scalability tests confirmed that rollout throughput increases near-linearly with the addition of compute nodes.

     Key Takeaways
     

6. Architectural Decoupling: ProRL AGENT treats the agentic rollout lifecycle as an independent HTTP service, segregating I/O-intensive tasks from GPU-intensive policy training.
7. Significant Performance Gains: The infrastructure enabled substantial performance improvements, with the Qwen3-8B model nearly doubling its performance on the SWE-Bench Verified benchmark.
8. System Latency Reductions: Targeted optimizations, such as replacing tmux with ptyprocess, contributed to near-linear throughput scaling across compute nodes.
9. Elimination of Tokenization Drift: The framework ensures that exact token IDs generated during rollout are passed to the trainer without the risk of lossy re-tokenization.
10. HPC-Native Deployment: By utilizing Singularity and supporting rootless execution with native Slurm integration, ProRL AGENT facilitates large-scale agent training on shared high-performance computing clusters.

    For more details, you can check out the Paper and Repo. Stay updated by following us on Twitter and joining our ML SubReddit.