We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks’ parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.

PDF

In Advances in Neural Information Processing Systems 34 (2020), Vancouver, Canada, December 2020.

@inproceedings{NeurIPS2020-Wu, title={Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks}, author={Lemeng Wu and Bo Liu and Peter Stone and Qiang Liu}, booktitle={Advances in Neural Information Processing Systems 34 (2020)}, month={December}, address={Vancouver, Canada}, url="http://www.cs.utexas.edu/users/ai-lab?NeurIPS2020-Wu", year={2020} }

Slides (PDF)