Candidate Name: Ashish Pujari
Program: Mechanical Engineering
Committee Chairs: Dr Scott D. Kelly
Committee Members: Dr Russell Keanini, Dr Tony Dear, Dr Artur Wolek, & Dr Omidreza Shoghli
Abstract:

Reinforcement learning (RL) has achieved immense success in developing optimal control strategies for various dynamical systems, ranging from classic board games like Go to applications in robotics, recommendation systems, and drug discovery. Despite this success, challenges remain --- most notably, the high computational cost of training a reinforcement learning agent on a complex system. In addition, sample inefficiency also remains one of the factors limiting RL's widespread adoption. Many notable techniques, such as experience replay, model-based learning, and imitation learning, have laid important groundwork to mitigate the issue around sample inefficiency, and in some capacity, the problem related to the computational burden. However, the sheer scale of real-world problems still prevents these methods from optimally addressing these issues. Additionally, even if one is able to solve the previously mentioned obstructions, the lack of global stability guarantees for RL-based controllers remains a critical concern. RL-based controllers are data-driven, with many state-of-the-art techniques utilising function approximators, such as neural networks, in their architecture. These lack the formal stability assurances required in safety-critical domains. This makes the usage of RL algorithms in high-stakes systems such as spacecraft and satellites very risky.

This thesis aims to address the aforementioned limitations by proposing methods that detect and exploit symmetries in the environment to mitigate sample inefficiency and reduce the computational cost of training. Additionally, it introduces a technique that leverages principles from stability theory to generate RL-based controllers with certified stability guarantees.

The first part of the study presents a novel approach for detecting and exploiting discrete group symmetries, particularly in discrete state-action spaces of environments used in tabular RL algorithms. Furthermore, experimental results demonstrating the effectiveness of this method across several benchmark environments are also presented.
Due to the increased complexity of continuous symmetries, the approach from the previous study does not directly extend to this setting. 

Consequently, the next part of this thesis addresses this challenge by incorporating tools from Riemannian geometry. In particular, a system exhibiting affine symmetries is used as benchmark to evaluate the efficacy of the proposed method.

Finally, the last part of this work presents a novel method for certifying the global stability of RL-based controllers. It addresses that gap by integrating concepts from Lyapunov stability theory and contraction analysis, introducing explicit stability constraints on the learnt policy.