UTCS Artificial Intelligence
courses
talks/events
demos
people
projects
publications
software/data
labs
areas
admin
Model-Based Exploration in Continuous State Spaces (2007)
Nicholas K. Jong
and
Peter Stone
Modern reinforcement learning algorithms effectively exploit experience data sampled from an unknown controlled dynamical system to compute a good control policy, but to obtain the necessary data they typically rely on naive exploration mechansisms or human domain knowledge. Approaches that first learn a model offer improved exploration in finite problems, but discrete model representations do not extend directly to continuous problems. This paper develops a method for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing model-based exploration mechanisms. Experiments with the resulting family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite data. This paper demonstrates instantiations of fitted-model algorithms that lead to faster learning on benchmark problems than contemporary model-free RL algorithms that only apply generalization in estimating action values. Finally, the paper concludes that in continuous problems, the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.
View:
PDF
,
PS
,
HTML
Citation:
In
The Seventh Symposium on Abstraction, Reformulation, and Approximation
, July 2007.
Bibtex:
@InProceedings{SARA07-jong, title={Model-Based Exploration in Continuous State Spaces}, author={Nicholas K. Jong and Peter Stone}, booktitle={The Seventh Symposium on Abstraction, Reformulation, and Approximation}, month={July}, url="http://www.cs.utexas.edu/users/ai-lab?SARA07-jong", year={2007} }
People
Nicholas Jong
Ph.D. Alumni
nickjong [at] me com
Peter Stone
Faculty
pstone [at] cs utexas edu
Areas of Interest
Other Areas
Reinforcement Learning
Labs
Learning Agents