• Classified by Topic • Classified by Publication Type • Sorted by Date • Sorted by First Author Last Name • Classified by Funding Source •
Decebal Constantin Mocanu, Elena
Mocanu, Peter Stone, Phuong
H. Nguyen, Madeleine Gibescu, and Antonio
Liotta. Evolutionary Training of Sparse Artificial Neural Networks: A Network Science Perspective. Technical Report
arXiv e-Prints 1707.04780, arXiv, 2017.
Available online
(unavailable)
Through the success of deep learning, Artificial Neural Networks (ANNs) are among the most used artificial intelligence methods nowadays. ANNs have led to major breakthroughs in various domains, such as particle physics, reinforcement learning, speech recognition, computer vision, and so on. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) Artificial Neural Networks (ANN), too, should not have fully-connected layers. We show how ANNs perform perfectly well with sparsely-connected layers. Following a Darwinian evolutionary approach, we propose a novel algorithm which evolves an initial random sparse topology (i.e. an Erd\Hos-Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during the ANN training process. The resulting sparse layers can safely replace the corresponding fully-connected layers. Our method allows to quadratically reduce the number of parameters in the fully conencted layers of ANNs, yielding quadratically faster computational times in both phases (i.e. training and inference), at no decrease in accuracy. We demonstrate our claims on two popular ANN types (restricted Boltzmann machine and multi-layer perceptron), on two types of tasks (supervised and unsupervised learning), and on 14 benchmark datasets. We anticipate that our approach will enable ANNs having billions of neurons and evolved topologies to be capable of handling complex real-world tasks that are intractable using state-of-the-art methods.
@TechReport{Decebal17,
author={Decebal Constantin Mocanu and Elena Mocanu and Peter Stone and Phuong H. Nguyen and Madeleine Gibescu and Antonio Liotta},
title={Evolutionary Training of Sparse Artificial Neural Networks: {A} Network Science Perspective},
year = "2017",
month = "August",
institution = "arXiv",
number = "arXiv e-Prints 1707.04780",
abstract="
Through the success of deep learning, Artificial Neural
Networks (ANNs) are among the most used artificial intelligence
methods nowadays. ANNs have led to major breakthroughs in
various domains, such as particle physics, reinforcement
learning, speech recognition, computer vision, and so
on. Taking inspiration from the network properties of
biological neural networks (e.g. sparsity, scale-freeness), we
argue that (contrary to general practice) Artificial Neural
Networks (ANN), too, should not have fully-connected layers. We
show how ANNs perform perfectly well with sparsely-connected
layers. Following a Darwinian evolutionary approach, we propose
a novel algorithm which evolves an initial random sparse
topology (i.e. an Erd\H{o}s-R\'enyi random graph) of two
consecutive layers of neurons into a scale-free topology,
during the ANN training process. The resulting sparse layers
can safely replace the corresponding fully-connected
layers. Our method allows to quadratically reduce the number of
parameters in the fully conencted layers of ANNs, yielding
quadratically faster computational times in both phases
(i.e. training and inference), at no decrease in accuracy. We
demonstrate our claims on two popular ANN types (restricted
Boltzmann machine and multi-layer perceptron), on two types of
tasks (supervised and unsupervised learning), and on 14
benchmark datasets. We anticipate that our approach will enable
ANNs having billions of neurons and evolved topologies to be
capable of handling complex real-world tasks that are
intractable using state-of-the-art methods. ",
wwwnote={<a href="https://arxiv.org/abs/1707.04780">Available online</a>},
}
Generated by bib2html.pl (written by Patrick Riley ) on Fri Sep 28, 2018 17:18:34