Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constra