Counterfactual Online Learning for Open-Loop Monte-Carlo Planning

Titelangaben

Phan, Thomy ; Chan, Shao-Hung ; Koenig, Sven:
Counterfactual Online Learning for Open-Loop Monte-Carlo Planning.
In: Proceedings of the AAAI Conference on Artificial Intelligence. Bd. 39 (2025) Heft 25 . - S. 26651-26658.
ISSN 2159-5399
DOI: https://doi.org/10.1609/aaai.v39i25.34867

Volltext

Link zum Volltext (externe URL):

Angaben zu Projekten

Abstract

Monte-Carlo Tree Search (MCTS) is a popular approach to online planning under uncertainty. While MCTS uses statistical sampling via multi-armed bandits to avoid exhaustive search in complex domains, common closed-loop approaches typically construct enormous search trees to consider a large number of potential observations and actions. On the other hand, open-loop approaches offer better memory efficiency by ignoring observations but are generally not competitive with closed-loop MCTS in terms of performance - even with commonly integrated human knowledge. In this paper, we propose Counterfactual Open-loop Reasoning with Ad hoc Learning (CORAL) for open-loop MCTS, using a causal multi-armed bandit approach with unobserved confounders (MABUC). CORAL consists of two online learning phases that are conducted during the open-loop search. In the first phase, observational values are learned based on preferred actions. In the second phase, counterfactual values are learned with MABUCs to make a decision via an intent policy obtained from the observational values. We evaluate CORAL in four POMDP benchmark scenarios and compare it with closed-loop and open-loop alternatives. In contrast to standard open-loop MCTS, CORAL achieves competitive performance compared with closed-loop algorithms while constructing significantly smaller search trees.