caa a22 4500 605514747 CHVBK 20210128100706.0 cr unu---uuuuu 210128e20150501xx s 000 0 eng 10.1007/s10458-014-9261-5 doi (NATIONALLICENCE)springer-10.1007/s10458-014-9261-5 Scalable solutions of interactive POMDPs using generalized and bounded policy iteration [Elektronische Daten] [Ekhlas Sonu, Prashant Doshi] Policy iteration algorithms for partially observable Markov decision processes (POMDPs) offer the benefits of quicker convergence compared to value iteration and the ability to operate directly on the solution, which usually takes the form of a finite state automaton. However, the finite state controller tends to grow quickly in size across iterations due to which its evaluation and improvement become computationally costly. Bounded policy iteration provides a way of keeping the controller size fixed while improving it monotonically until convergence, although it is susceptible to getting trapped in local optima. Despite these limitations, policy iteration algorithms are viable alternatives to value iteration, and allow POMDPs to scale. In this article, we generalize the bounded policy iteration technique to problems involving multiple agents. Specifically, we show how we may perform bounded policy iteration with anytime behavior in settings formalized by the interactive POMDP framework, which generalizes POMDPs to non-stationary contexts shared with multiple other agents. Although policy iteration has been extended to decentralized POMDPs, the context there is strictly cooperative. Its novel generalization in this article makes it useful in non-cooperative settings as well. As interactive POMDPs involve modeling other agents sharing the environment, we ascribe controllers to predict others' actions, with the benefit that the controllers compactly represent the model space. We show how we may exploit the agent's initial belief, often available, toward further improving the controller, particularly in large domains, though at the expense of increased computations, which we compensate. We extensively evaluate the approach on multiple problem domains with some that are significantly large in their dimensions, and in contexts with uncertainty about the other agent's frames and those involving multiple other agents, and demonstrate its properties and scalability. The Author(s), 2014 Decision making nationallicence Multiagent settings nationallicence Policy iteration nationallicence POMDP nationallicence Sonu Ekhlas Department of Computer Science, University of Georgia, 30602, Athens, GA, USA aut Doshi Prashant Department of Computer Science, University of Georgia, 30602, Athens, GA, USA aut Autonomous Agents and Multi-Agent Systems Springer US; http://www.springer-ny.com 29/3(2015-05-01), 455-494 1387-2532 29:3<455 2015 29 10458 https://doi.org/10.1007/s10458-014-9261-5 text/html Onlinezugriff via DOI BK010053 XK010053 XK010000 Metadata rights reserved Springer special CC-BY-NC licence nationallicence 1 research-article jats NATIONALLICENCE NATIONALLICENCE NL-springer NATIONALLICENCE 856 40 https://doi.org/10.1007/s10458-014-9261-5 text/html Onlinezugriff via DOI NATIONALLICENCE 700 1- Sonu Ekhlas Department of Computer Science, University of Georgia, 30602, Athens, GA, USA aut NATIONALLICENCE 700 1- Doshi Prashant Department of Computer Science, University of Georgia, 30602, Athens, GA, USA aut NATIONALLICENCE 773 0- Autonomous Agents and Multi-Agent Systems Springer US; http://www.springer-ny.com 29/3(2015-05-01), 455-494 1387-2532 29:3<455 2015 29 10458