pipette
ESEspañol

Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample

Wenye Li, Ke Wei

Preprint

In the authors' words

Policy mirror descent (PMD) refers to a class of policy optimization methods associated with different mirror maps. For a broad class of decomposable Legendre mirror maps encompassing many standard choices, Li and Wei [2026] show that the PMD policy sequence converges to an optimal policy from any admissible initialization under any positive constant step size, even when the optimal policy is not unique. In this note, we provide a counterexample with assistance from OpenAI's GPT-5.6 model, showing that PMD with a non-decomposable Legendre mirror map need not converge in policy.

Main resultThe abstract does not state a limitation.

Appeared: Thursday, October 15. arXiv. Preprint, not yet peer-reviewed.