UTCS Artificial Intelligence
courses
talks/events
demos
people
projects
publications
software/data
labs
areas
admin
Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks (2020)
Lemeng Wu, Bo Liu,
Peter Stone
, and Qiang Liu
We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks’ parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.
View:
PDF
Citation:
In
Advances in Neural Information Processing Systems 34 (2020)
, Vancouver, Canada, December 2020.
Bibtex:
@inproceedings{NeurIPS2020-Wu, title={Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks}, author={Lemeng Wu and Bo Liu and Peter Stone and Qiang Liu}, booktitle={Advances in Neural Information Processing Systems 34 (2020)}, month={December}, address={Vancouver, Canada}, url="http://www.cs.utexas.edu/users/ai-lab?NeurIPS2020-Wu", year={2020} }
Presentation:
Slides (PDF)
People
Peter Stone
Faculty
pstone [at] cs utexas edu
Areas of Interest
Machine Learning
Labs
Learning Agents