Implement pass rate-based curriculum learning with weighted sampling

[jb3618columbia]

Summary

Implements curriculum learning using pass rate-based weighted sampling for GRPO training

Changes

• Add a PassRateTracker class to track attempt and success counts for each prompt. This tracker can be used for multiple curriculum samplers
• Add a PassRateWeightedSampler class which implements a weighted sampler that adjusts sampling probabilities (probability of prompts sampled in a batch) based on historical pass rates (optional to use exponential moving average)
• Update DAPOTrainer: pass rate tracker during training and logs curriculum metrics
• Minor edits to make the integration work

Testing

• Tested with local single-node runs

• Tested with multi-node SLURM runs (2 nodes, 8 GPUs each)

• Logs curriculum metrics: hardest_10pct/25pct/50pct/75pct pass rates, batch-level statistics

• See curriculum learning runs: https://wandb.ai/mbzuai-llm/Reasoning360/runs/qab27nv0?nw=nwuserjalajbhandari

• Example run: Curriculum-1435219-qwen2.5-32b-base-fsdp-temp_0.5_data_mixtures_round2_train_prompt_bsz_32

1. Effective batch size increases with training, decreases when hard samples get sampled and then increases as the model learns to solve hard problems

2. Pass rates of hard examples increase with training: model focuses on harder problems and starts to solve these (tracking the percentile of hard problems based on historical pass rates)

3. Right skew to the count distribution which shows the some prompts are only attempted a few times (easy prompts) while other samples are attempted multiple times (hard prompts)

[Copilot]

Pull request overview

This PR implements pass rate-based curriculum learning for GRPO training by introducing weighted sampling that prioritizes harder samples (those with lower historical success rates).

Changes:

• Added PassRateTracker class to track attempt counts and success rates for each prompt in the dataset
• Added PassRateWeightedSampler class that implements curriculum learning through dynamic weighted sampling based on historical pass rates
• Integrated curriculum learning into the DAPO trainer with pass rate tracking and curriculum-specific metrics logging
• Updated configuration files and training scripts with curriculum learning examples

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
verl/utils/pass_rate_tracker.py	Core tracker for maintaining historical pass rates and attempt counts per sample
verl/utils/pass_rate_weighted_sampler.py	Weighted sampler that adjusts sampling probabilities based on pass rates
verl/utils/dataset/rl_dataset.py	Added dataset_index field to enable sample tracking
verl/trainer/ppo/ray_trainer.py	Added comment clarifying sampler creation
verl/trainer/ppo/metric_utils.py	Added reward standard deviation metric
verl/trainer/config/data/legacy_data.yaml	Added curriculum sampler configuration parameters
recipe/dapo/dapo_ray_trainer.py	Integrated pass rate tracking and curriculum metrics logging into training loop
scripts/*	Added example training scripts demonstrating curriculum learning usage

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nightlessbaron]

Hey @jb3618columbia , this is good start. However, there are lots of things that we can improve on:

1. Can we add customization to it such that the user can add a custom strategy in the weight sampler?
2. Also add customization such that the user can define how many steps to wait before updating the weights.
3. Please add some tests or optionally you can add them to RL360.
4. Please add a short documentation as a quick start guide for people to use.
5. Address all of my comments as well as those from codex :)

Good job overall :D

[nightlessbaron]

remove this file from your changes

[jb3618columbia]

Do we want to keep the changes we made to the cluster environment set up so that people in the future can use the example script? If not, I can remove all the above changes.

One thought is to update/clean up these files as we go along so people can refer/use these out of the box

[nightlessbaron]

as discussed, let's remove them for now

[nightlessbaron]

remove this file from your changes

[nightlessbaron]

not sure i understand this

[jb3618columbia]

My understanding is that having dataloader_num_workers > 0 will not be able to sample batches from the latest set of weights becuase of caching, but perhaps I am incorrect here

[nightlessbaron]

It just means the DataLoader will use multiple worker subprocesses to fetch and preprocess batches in parallel with your training loop. If you set it to 0, data loading would happen in the main training process.

[jb3618columbia]

Hmm, would it lead to ''off policyness'' in terms of the weighting distribution? That was my concern

[jb3618columbia]

@nightlessbaron this config is a now used for pass rate based weighted sampling