VAST: Value Function Factorization with Variable Agent Sub-Teams

Title data

Phan, Thomy ; Ritz, Fabian ; Belzner, Lenz ; Altmann, Philipp ; Gabor, Thomas ; Linnhoff-Popien, Claudia:
VAST: Value Function Factorization with Variable Agent Sub-Teams.
In: Ranzato, Marc'Aurelio ; Beygelzimer, A. ; Dauphin, Y. ; Liang, P. S. ; Vaughan, J. Wortman (ed.): Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021). - Red Hook, NY : Curran Associates, Inc. , 2021 . - pp. 24018-24032 . - (Advances in Neural Information Processing Systems ; 34 )
ISBN 978-1-7138-4539-3

Project information

Abstract in another language

Value function factorization (VFF) is a popular approach to cooperative multi-agent reinforcement learning in order to learn local value functions from global rewards. However, state-of-the-art VFF is limited to a handful of agents in most domains. We hypothesize that this is due to the flat factorization scheme, where the VFF operator becomes a performance bottleneck with an increasing number of agents. Therefore, we propose VFF with variable agent sub-teams (VAST). VAST approximates a factorization for sub-teams which can be defined in an arbitrary way and vary over time, e.g., to adapt to different situations. The sub-team values are then linearly decomposed for all sub-team members. Thus, VAST can learn on a more focused and compact input representation of the original VFF operator. We evaluate VAST in three multi-agent domains and show that VAST can significantly outperform state-of-the-art VFF, when the number of agents is sufficiently large.