A Concentration Bound for Two-Timescale Actor-Critic Algorithm

Prashansa Panda; Shalabh Bhatnagar

In the authors' words

Significant research effort has been directed in recent years towards establishing both asymptotic and non-asymptotic convergence guarantees for two-timescale actor--critic algorithms, where the actor recursion is run on a slower timescale than the critic recursion. This work derives a uniform all-time concentration bound for the actor--critic algorithm with function approximation in the long-run average-reward setting. This bound helps us analyze the behavior of the actor parameter with high probability. We show that, after some finite time, the actor parameter enters a safe region and remains within it thereafter with high probability. Specifically, with probability at least $1 - ϵ_{1} - ϵ_{2}$ , the actor error $∥ θ_{k} - θ^{*} ∥$ is $O (\frac{n _{0}^{3/4}}{k} \frac{1}{ϵ _{2}} + (\frac{1}{n _{0}})^{1/4} lo g^{1/4} (\frac{1}{ϵ _{1}}) + (\frac{1}{n _{0}})^{1/4})$ for all $k \geq n_{0}$ and sufficiently large $n_{0}$ . We also present experimental results demonstrating that the aforementioned actor error diminishes with the number of actor-parameter updates.

Main resultThe abstract does not state a limitation.

A Concentration Bound for Two-Timescale Actor-Critic Algorithm

In the authors' words

More on Reinforcement learning that day

Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning

Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents

A General Framework for Budgeted Threshold Incentives on Request

RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory

Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management