UTCS Colloquia - Quoc Le, Faculty Candidate, Stanford University, "Scaling deep learning to 10,000 cores and beyond" ACE 2.302

Contact Name: 
Kate Callard
ACES 2.302
Mar 26, 2013 11:00am - 12:00pm

Signup Schedule: http://apps.cs.utexas.edu/talkschedules/cgi/list_events.cgi

Talk Audience: UTCS Faculty, Grads, Undergrads, Other Interested Parties

Host:  Inderjit S Dhillon

Talk Abstract: Deep learning and unsupervised feature learning offer the potential to transform many domains such as vision, speech, and natural language processing.  However, these methods have been fundamentally limited by our computational abilities, and typically applied to small-sized problems.  In this talk, I describe the key ideas that enabled scaling deep learning algorithms to train a large model on a cluster of 16,000 CPU cores (2000 machines).  This network has 1.15 billion parameters, which is more than 100x larger than the next largest network reported in the literature.

Such network, when applied at the huge scale, is able to learn abstract concepts in a much more general manner than previously demonstrated. Specifically, we find that by training on 10 million unlabeled images, the network produces features that are very selective for high-level concepts such as human faces and cats. Using these features, we also obtain breakthrough performance gains on several large-scale computer vision tasks. 

Thanks to its scalability and insensitivity to modalities, our framework is also used successfully to achieve performance leaps in other domains, such as speech recognition and natural language understanding.

Speaker Bio: Quoc V. Le is a PhD student at Stanford and a software engineer at Google. At Stanford and Google, Quoc works on large scale brain simulation using unsupervised feature learning and deep learning. His recent work in deep learning and big data, that yields state-of-the-art performances in many pattern recognition tasks, was widely distributed and discussed on various technology blogs and news sites. Quoc obtained his undergraduate degree with First Class Honours and Distinguished Scholar at the Australian National University. During his undergraduate, he worked on kernel methods, ranking, structured output prediction and optimization. His undergraduate honours thesis work won best paper award as ECML 2007. Quoc was also a researcher at National ICT Australia, Microsoft Research and Max Planck Institute of Biological Cybernetics.