Peter Stone's Selected Publications

Classified by TopicClassified by Publication TypeSorted by DateSorted by First Author Last NameClassified by Funding Source


Empowerment for continuous agent-environment systems

Empowerment for continuous agent-environment systems.
Tobias Jung, Daniel Polani, and Peter Stone.
Adaptive Behavior, 19(1):16–39, 2011.
Available from Adaptive Behavior page.

Download

[PDF]1.8MB  [postscript]1.8MB  

Abstract

This paper develops generalizations of empowerment to continuous states. Empowerment is a recently introduced information-theoretic quantity motivated by hypotheses about the efficiency of the sensorimotor loop in biological organisms, but also from considerations stemming from curiosity-driven learning. Empowemerment measures, for agent-environment systems with stochastic transitions, how much influence an agent has on its environment, but only that influence that can be sensed by the agent sensors. It is an information-theoretic generalization of joint controllability (influence on environment) and observability (measurement by sensors) of the environment by the agent, both controllability and observability being usually defined in control theory as the dimensionality of the control/observation spaces. Earlier work has shown that empowerment has various interesting and relevant properties, e.g., it allows us to identify salient states using only the dynamics, and it can act as intrinsic reward without requiring an external reward. However, in this previous work empowerment was limited to the case of small-scale and discrete domains and furthermore state transition probabilities were assumed to be known. The goal of this paper is to extend empowerment to the significantly more important and relevant case of continuous vector-valued state spaces and initially unknown state transition probabilities. The continuous state space is addressed by Monte-Carlo approximation; the unknown transitions are addressed by model learning and prediction for which we apply Gaussian processes regression with iterated forecasting. In a number of well-known continuous control tasks we examine the dynamics induced by empowerment and include an application to exploration and online model-learning.

BibTeX Entry

@Article{AB11-jung,
 author="Tobias Jung and Daniel Polani and Peter Stone",
 title="Empowerment for continuous agent-environment systems",
 journal="Adaptive Behavior",
 volume="19",
 number="1",
 pages="16-39",
 year="2011",
 abstract={
           This paper develops generalizations of empowerment to
           continuous states. Empowerment is a recently introduced
           information-theoretic quantity motivated by hypotheses
           about the efficiency of the sensorimotor loop in biological
           organisms, but also from considerations stemming from
           curiosity-driven learning. Empowemerment measures, for
           agent-environment systems with stochastic transitions, how
           much influence an agent has on its environment, but only
           that influence that can be sensed by the agent sensors. It
           is an information-theoretic generalization of joint
           controllability (influence on environment) and
           observability (measurement by sensors) of the environment
           by the agent, both controllability and observability being
           usually defined in control theory as the dimensionality of
           the control/observation spaces.  Earlier work has shown
           that empowerment has various interesting and relevant
           properties, e.g., it allows us to identify salient states
           using only the dynamics, and it can act as intrinsic reward
           without requiring an external reward. However, in this
           previous work empowerment was limited to the case of
           small-scale and discrete domains and furthermore state
           transition probabilities were assumed to be known. The goal
           of this paper is to extend empowerment to the significantly
           more important and relevant case of continuous
           vector-valued state spaces and initially unknown state
           transition probabilities. The continuous state space is
           addressed by Monte-Carlo approximation; the unknown
           transitions are addressed by model learning and prediction
           for which we apply Gaussian processes regression with
           iterated forecasting. In a number of well-known continuous
           control tasks we examine the dynamics induced by
           empowerment and include an application to exploration and
           online model-learning.},
  wwwnote={Available from <a href="http://adb.sagepub.com/content/19/1/16.abstract">Adaptive Behavior</a> page.},
} 

Generated by bib2html.pl (written by Patrick Riley ) on Mon Mar 25, 2024 00:05:10