Temporal difference methods are theoretically grounded and empirically effective methods for addressing reinforcement learning problems. In most real-world reinforcement learning tasks, TD methods require a function approximator to represent the value function. However, using function approximators requires manually making crucial representational decisions. This paper investigates emphevolutionary function approximation, a novel approach to automatically selecting function approximator representations that enable efficient individual learning. This method emphevolves individuals that are better able to emphlearn. We present a fully implemented instantiation of evolutionary function approximation which combines NEAT, a neuroevolutionary optimization technique, with Q-learning, a popular TD method. The resulting NEAT+Q algorithm automatically discovers effective representations for neural network function approximators. This paper also presents emphon-line evolutionary computation, which improves the on-line performance of evolutionary computation by borrowing selection mechanisms used in TD methods to choose individual actions and using them in evolutionary computation to select policies for evaluation. We evaluate these contributions with extended empirical studies in two domains: 1) the mountain car task, a standard reinforcement learning benchmark on which neural network function approximators have previously performed poorly and 2) server job scheduling, a large probabilistic domain drawn from the field of autonomic computing. The results demonstrate that evolutionary function approximation can significantly improve the performance of TD methods and on-line evolutionary computation can significantly improve evolutionary methods. This paper also presents additional tests that offer insight into what factors can make neural network function approximation difficult in practice.

PDF, PS, HTML

Journal of Machine Learning Research, Vol. 7 (2006), pp. 877-917.

@Article{whiteson:jmlr2006, title={Evolutionary Function Approximation for Reinforcement Learning}, author={Shimon Whiteson and Peter Stone}, volume={7}, journal={Journal of Machine Learning Research}, month={May}, pages={877-917}, url="http://www.cs.utexas.edu/users/ai-lab?whiteson:jmlr2006", year={2006} }