Peter Stone's Selected Publications

Classified by TopicClassified by Publication TypeSorted by DateSorted by First Author Last NameClassified by Funding Source


Model-Selection for Non-Parametric Function Approximation in Continuous Control Problems: A Case Study in a Smart Energy System

Daniel Urieli and Peter Stone. Model-Selection for Non-Parametric Function Approximation in Continuous Control Problems: A Case Study in a Smart Energy System. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML'13), Sep 2013.
Official publisher version

Download

[PDF]2.3MB  [postscript]1.4MB  

Abstract

This paper investigates the application of value-function-based reinforcement learning to a smart energy control system, specifically the task of controlling an HVAC system to minimize energy while satisfying residents' comfort requirements. In theory, value-function-based reinforcement learning methods can solve control problems such as this one optimally. However, since choosing an appropriate parametric representation of the value function turns out to be difficult, we develop an alternative method, which results in a practical algorithm for value function approximation in continuous state-spaces. To avoid the need to carefully design a parametric representation for the value function, we use a smooth non-parametric function approximator, specifically Locally Weighted Linear Regression (LWR). LWR is used within Fitted Value Iteration (FVI), which has met with several practical successes. However, for efficiency reasons, LWR is used with a limited sample-size, which leads to poor performance without careful tuning of LWR's parameters. We therefore develop an efficient meta-learning procedure that performs online model-selection and tunes LWR's parameters based on the Bellman error. Our algorithm is fully implemented and tested in a realistic simulation of the HVAC control domain, and results in significant energy savings.

BibTeX Entry

@InProceedings{ECML13-urieli,
  author = {Daniel Urieli and Peter Stone},
  title = {Model-Selection for Non-Parametric Function Approximation in Continuous Control Problems: A Case Study in a Smart Energy System},
  booktitle = {Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML'13)},
  location = {Prague, Czech Republic},
  month = {Sep},
  year = {2013},
  abstract = "
          This paper investigates the application of value-function-based reinforcement
          learning to a smart energy control system, specifically the task of
          controlling an HVAC system to minimize energy while satisfying residents'
          comfort requirements.  In theory, value-function-based reinforcement learning
          methods can solve control problems such as this one optimally.  However,
          since choosing an appropriate parametric representation of the value function
          turns out to be difficult, we develop an alternative method, which results in
          a practical algorithm for value function approximation in continuous
          state-spaces.  To avoid the need to carefully design a parametric
          representation for the value function, we use a smooth non-parametric
          function approximator, specifically Locally Weighted Linear Regression (LWR).
          LWR is used within Fitted Value Iteration (FVI), which has met with several
          practical successes.  However, for  efficiency reasons, LWR is used with a
          limited sample-size, which leads to poor performance without careful tuning
          of LWR's parameters.  We therefore develop an efficient meta-learning
          procedure that performs online model-selection and tunes LWR's parameters
          based on the Bellman error.  Our algorithm is fully implemented and tested in
          a realistic simulation of the HVAC control domain, and results in significant
          energy savings.  
  ",
  wwwnote={Official <a href="http://link.springer.com/chapter/10.1007/978-3-642-40988-2_5">publisher version</a>},
}

Generated by bib2html.pl (written by Patrick Riley ) on Thu Dec 11, 2014 23:22:57