Peter Stone's Selected Publications

Classified by TopicClassified by Publication TypeSorted by DateSorted by First Author Last NameClassified by Funding Source


Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic Implementations for Reproducibility in Deep Reinforcement Learning. In 2nd Reproducibility in Machine Learning Workshop at ICML 2018, July 2018.

Download

[PDF]6.6MB  

Abstract

While deep reinforcement learning (DRL) has led to numerous successes in recentyears, reproducing these successes can be extremely challenging. Onereproducibility challenge particularly relevant to DRL is nondeterminism in thetraining process, which can substantially affect the results. Motivated by thischallenge, we study the positive impacts of deterministic implementations ineliminating nondeterminism in training. To do so, we consider the particularcase of the deep Q-learning algorithm, for which we produce a deterministicimplementation by identifying and controlling all sources of nondeterminism inthe training process. One by one, we then allow individual sources ofnondeterminism to affect our otherwise deterministic implementation, andmeasure the impact of each source on the variance in performance. We find thatindividual sources of nondeterminism can substantially impact the performanceof agent, illustrating the benefits of deterministic implementations. Inaddition, we also discuss the important role of deterministic implementationsin achieving exact replicability of results.

BibTeX Entry

@InProceedings{RML18-nagarajan,
  author = {Prabhat Nagarajan and Garrett Warnell and Peter Stone},
  title = {Deterministic Implementations for Reproducibility in Deep Reinforcement Learning},
  booktitle = {2nd Reproducibility in Machine Learning Workshop at ICML 2018},
  location = {Stockholm, Sweden},
  month = {July},
  year = {2018},
  abstract = {
While deep reinforcement learning (DRL) has led to numerous successes in recent
years, reproducing these successes can be extremely challenging. One
reproducibility challenge particularly relevant to DRL is nondeterminism in the
training process, which can substantially affect the results. Motivated by this
challenge, we study the positive impacts of deterministic implementations in
eliminating nondeterminism in training. To do so, we consider the particular
case of the deep Q-learning algorithm, for which we produce a deterministic
implementation by identifying and controlling all sources of nondeterminism in
the training process. One by one, we then allow individual sources of
nondeterminism to affect our otherwise deterministic implementation, and
measure the impact of each source on the variance in performance. We find that
individual sources of nondeterminism can substantially impact the performance
of agent, illustrating the benefits of deterministic implementations. In
addition, we also discuss the important role of deterministic implementations
in achieving exact replicability of results.
  },
}

Generated by bib2html.pl (written by Patrick Riley ) on Fri Jun 14, 2019 11:36:28