caa a22 4500 605514739 CHVBK 20210128100706.0 cr unu---uuuuu 210128e20150501xx s 000 0 eng 10.1007/s10458-014-9257-1 doi (NATIONALLICENCE)springer-10.1007/s10458-014-9257-1 Hernández-Orallo José DSIC, Universitat Politècnica de València, Valencia, Spain aut On environment difficulty and discriminating power [Elektronische Daten] [José Hernández-Orallo] This paper presents a way to estimate the difficulty and discriminating power of any task instance. We focus on a very general setting for tasks: interactive (possibly multi-agent) environments where an agent acts upon observations and rewards. Instead of analysing the complexity of the environment, the state space or the actions that are performed by the agent, we analyse the performance of a population of agent policies against the task, leading to a distribution that is examined in terms of policy complexity. This distribution is then sliced by the algorithmic complexity of the policy and analysed through several diagrams and indicators. The notion of environment response curve is also introduced, by inverting the performance results into an ability scale. We apply all these concepts, diagrams and indicators to two illustrative problems: a class of agent-populated elementary cellular automata, showing how the difficulty and discriminating power may vary for several environments, and a multi-agent system, where agents can become predators or preys, and may need to coordinate. Finally, we discuss how these tools can be applied to characterise (interactive) tasks and (multi-agent) environments. These characterisations can then be used to get more insight about agent performance and to facilitate the development of adaptive tests for the evaluation of agent abilities. The Author(s), 2014 Environment difficulty nationallicence Agent evaluation nationallicence Discriminating power nationallicence Agent policy nationallicence Algorithmic information theory nationallicence Universal psychometrics nationallicence Reinforcement learning nationallicence Elementary cellular automata nationallicence Autonomous Agents and Multi-Agent Systems Springer US; http://www.springer-ny.com 29/3(2015-05-01), 402-454 1387-2532 29:3<402 2015 29 10458 https://doi.org/10.1007/s10458-014-9257-1 text/html Onlinezugriff via DOI BK010053 XK010053 XK010000 Metadata rights reserved Springer special CC-BY-NC licence nationallicence 1 research-article jats NATIONALLICENCE NATIONALLICENCE NL-springer NATIONALLICENCE 856 40 https://doi.org/10.1007/s10458-014-9257-1 text/html Onlinezugriff via DOI NATIONALLICENCE 100 1- Hernández-Orallo José DSIC, Universitat Politècnica de València, Valencia, Spain aut NATIONALLICENCE 773 0- Autonomous Agents and Multi-Agent Systems Springer US; http://www.springer-ny.com 29/3(2015-05-01), 402-454 1387-2532 29:3<402 2015 29 10458