RL-projects

The collection of all my practiced RL projects

• Stages A-B: RL mechanics + tabular RL – 6 levels

  • Level 1: Two-Armed Bandit
  • Level 2: Episodic 1D GridWorld with a random policy
  • Level 3: Monte Carlo State-Value Estimation
  • Level 4: Monte Carlo Action-Value Estimation and Policy Improvement
  • Level 5: Q-Learning: Learning from One Step at a Time
  • Level 6: SARSA: On-Policy Temporal-Difference Control

• Stage C: Gymnasium – 2 levels

  • Level 7: Gymnasium: Using a Standard RL Environment
  • Level 8: Q-Learning with Gymnasium: FrozenLake

• Stage D: Neural-network RL basics – 3 levels

  • Level 9: Neural Stochastic Policy with PyTorch
  • Level 10: One REINFORCE Policy Update
  • Level 11: Train REINFORCE Across Episodes

• Stage E: Policy gradients / REINFORCE – 2 levels

  • Level 12: REINFORCE with a Learned Value Baseline
  • Level 13: REINFORCE with an Entropy Bonus

• Stage F: Actor-Critic + PPO – 9 levels

  • Level 14: Online Actor–Critic with One-Step TD Error
  • Level 15: Generalized Advantage Estimation (GAE)
  • Level 16: PPO Probability Ratios and Clipped Policy Updates
  • Level 17: Fixed-Length PPO Rollouts and Boundary-Aware GAE
  • Level 18: PPO with Shuffled Minibatch Updates
  • Level 19: PPO with Vectorized Environments
  • Level 20: PPO KL Diagnostics and Early Stopping
  • Level 21: PPO Gradient Clipping

• Stage G: RL → LLM transition – ~2-3 levels

• Stage H: LLM post-training tooling – ~2 levels

• Stage I: RLHF / RLAIF / RLVR – ~2 levels

• Stage J: actual miniature RLVR fine-tuning – ~3-4 levels

• Stage K: research/thesis-level code – ongoing