Deep reinforcement learning methods

Abstract

This thesis focuses on Deep Reinforcement Learning (Deep RL) methods and their applications in games. It provides an in-depth analysis of existing state-of-the-art Deep RL methods and introduces novel techniques based on the concept of meta-RL for achieving high performance with improved sample-efficiency. Along with the theoretical foundations of Machine Learning (ML) and RL, game AI applications are also considered. This chapter summarizes the presented contributions and discusses future directions of the work on current open issues in the field. Τhe first contribution of this thesis includes a comprehensive overview of Deep RL and state-of-the-art algorithms that have achieved impressive results on various platforms and environments. The analysis delves deeply into these models, offering valuable insights and proposing new implementations that combine core Deep RL algorithms with different host algorithms. The functionalities and properties of each algorithm are also thoroughly examined, highlighting both their common and unique elements, and discussing their suitability for research and development purposes. A detailed comparative analysis of reported results from published works is presented, providing a comprehensive view of their performance on the aforementioned platforms. This analysis includes both quantitative and qualitative results, providing insights regarding the future directions in the field of Deep RL. Next, a proposed meta-RL methodology is presented, termed REIN-2, which is a novel end-to-end Deep RL framework designed to address mainly the critical problem of sample-inefficiency in Deep RL algorithms. For this meta-learning scheme, a pair of Deep RL algorithms are employed and used in conjunction for improved performance and sample-efficiency. This thesis also presents GENEREIT, an extension of the REIN-2 methodology, developed to address the problem of generalization in Deep RL models as well. Performance measurements and insights are shown by conducting different sets of experiments, using various environments. Additionally, in the core of this thesis' contributions are various ML applications in (video) games. More specifically, we present Lucy-SKG, a Deep RL-based bot designed to play the commercial game of Rocket League. Through various proposed novelties, Lucy-SKG outperforms Necto, the 2022 Rocket League Bot Championship winner, by a significant margin. Next, AlphaBluff is presented, which is a modern 3D Heads-Up Texas Hold'em Poker video game that incorporates AI models that were implemented and trained, where human players can play against them. For this application, several algorithms were explored, analyzed, implemented, trained and compared in terms of performance and characteristics, with statistical reports indicating distinctive performance against human players. Following, ADHD360 is presented, which is a serious game developed for the purposes of diagnosing Attention Deficit Hyperactivity Disorder (ADHD) in human players using supervised ML methods. The serious game The Delivery is also presented in this thesis, which was developed for the purpose of diagnosing Major Depressive Disorder in human players using supervised ML methods. The last of the contributions presented in this thesis corresponds to Mind Escape, an Escape Room video game developed with the purpose of analyzing human players’ behavior and identifying their prominent cognitive features and personality traits using Deep RL methods.

Το αντικείμενο της παρούσας διατριβής είναι οι μέθοδοι Βαθιάς Ενισχυτικής Μάθησης (Deep Reinforcement Learning). Η διατριβή εξερευνά υπάρχουσες τεχνικές Βαθιάς Ενισχυτικής Μάθησης, ενώ επίσης παρουσιάζει νέες, οι οποίες έχουν βελτιωμένες επιδόσεις σε κοινά περιβάλλοντα αξιολόγησης σε σύγκριση με μεθόδους αιχμής της σχετικής βιβλιογραφίας, ενώ ταυτόχρονα βελτιώνουν συγκεκριμένες πτυχές τους, και πιο συγκεκριμένα, τη δειγματική αποδοτικότητα (sample-efficiency) και την ικανότητα γενίκευσης (generalization). Τέλος, παρουσιάζονται διάφορες εφαρμογές τεχνικών Μηχανικής Μάθησης σε παιχνίδια.