FLARE

End-to-end Memory Networks

Slides

Continuous control with deep reinforcement learning

Second reading. Slides.

Human-level control through deep reinforcement learning

First Reading

WaveNet: A Generative Model for Raw Audio

Modern paper

Pixel Recurrent Neural Networks

Classic paper

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Modern paper Slides

Generative Adversarial Nets

Classic paper: GANs. Slides

Modern paper: Variational Inference with Normalizing Flows

Presented by Wesley Slides

Auto-Encoding Variational Bayes

Classic paper: AE VB

Layer Normalization

Modern paper: Layer Norm

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Classic paper: Batchnorm

Deep Residual Learning for Image Recognition

Resnet

Recurrent Batch Normalization

SR: Apply BN after *every* timestep in RNN. Works dramatically.

Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

SR: they present a deep-Q+LSTM which is able to solve some classic riddles. First combination of deep-Q reinforcement with an LSTM that I've seen

Net2Net: Accelerating Learning via Knowledge Transfer

Dinesh: On the lines of the "Model Compression" idea, but now trying to transfer information from a small easy-to-train network to accelerate the training of a larger network

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Eddie: General NN framework for explicitly representing input as a knowledge base that can be iterated over multiple times. Motivated by question-answering and transitive reasoning. General enough to be instantiated for many different tasks.

Mastering the game of Go with deep neural networks and tree search

SR: First AI to beat a professional go player

Teaching Deep Convolutional Neural Networks to Play Go

Matthew: Cool looking paper outlining a neural network approach to a game has been dominated by UCT planning.

Do Deep Nets Really Need to be Deep?

Karl: An extended abstract (4pp) providing empirical evidence that shallow nets can do as well as deep nets with the same number of parameters

Neural Turing Machines

SR: Fresh off the press.

Why Does Unsupervised Pre-training Help Deep Learning?

Craig: An in-depth discussion of the title question, with experiments that illuminate what is going on in an intuitive way. The short answer seems to be data-dependent regularization.

Multimodal Deep Learning with Deep Boltzman Machine

Aditya: Interesting to read how it combines sparse word frequencies from text with SIFT features from images. Since it builds generative models on text and image features, it can work even when data from one of input sources is missing.

Visualizing and Understanding Convolutional Networks

Matthew: Presents a method for visualizing the different layers in convolutional networks. Analyzes and improves upon Krizevsky's Imagenet network.

Maxout networks

Leif: Apparently a maxout unit is a piecewise linear combination of its inputs, combining some of the benefits of rectified linear activations with dropouts (?)

Provably Efficient Algorithm for Training Deep Networks

Aditya: Proposes a principled way of learning deep polynomial features. It works well on small dataset like MNIST with minimal parameter tuning. I am evaluating this for larger datasets.

Efficient Estimation of Word Representations in Vector Space

Aditya: Recent paper from Google folks (along with code) where they constructed high level vector representations of text. These features can help find semantic similarities in words.

Generating text with recurrent neural networks

Leif: This paper describes a recurrent network that was trained on character sequences from Wikipedia, and is capable of generating character sequences that have remarkably sophisticated structure. Also interesting because the model includes multiplicative interactions.

Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

Craig: Proposes a new way of regularizing autoencoders: an analytical penalty on the Jacobian of the encoder activations wrt the input. This is motivated by and related to the denoising auto-encoder, which has had empirical success.

Multiscale scattering for audio classification

Leif: Developed by Stephane Mallat (creator of Matching Pursuit and a wavelet pioneer), this paper describes a "deep" wavelet transform that provides a nice representation for audio data classification. Particularly interesting is the use of features at all levels simultaneously.

Playing Atari with Deep Reinforcement Learning

Leif: A deep model is used to map from pixels in atari games to some sort of Q-learning-like reward signal.

A Deep Non-Linear Feature Mapping for Large-Margin kNN Classification

Elad: Using deep models to enhance kNN

Learning Deep Energy Models

Craig: Presents an interesting alternative to learning deep probabilistic models: only let the top layer be probabilistic and all lower layers create a deterministic feed-forward neural net. This avoids complicated sampling or variational techniques used to train previous deep probabilistic models (ex. DBN,DBM)

Modeling Pixel Means and Covariances Using Factorized Third-Order Boltzmann Machines

Leif: An important addition to the canon of RBM / energy-based models. This model (mcRBM) explicitly captures the mean and the covariance of the visible units, and gives good results on a wide variety of datasets.

A fast learning algorithm for deep belief networks

Leif: A foundational paper that opened up the "deep" learning field. This paper puts a nice probabilistic background on deep architectures based on RBMs.

Representation Learning: A Review and New Perspectives

Craig: Quite a long one, but a good overview of a field that is changing rapidly by Bengio, one of the leaders in the field. Might be a good one to start off with or have here as background reading.

Neural GPUs Learn Algorithms

Karl: Like Neural Turing Machines, but parallelizable

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Karl: Usually, when we train sequence-decoders, we feed the gold-standard output as the next step's input. This is KIND OF strange since the model doesn't necessarily know how to generate the gold output yet, so there's a mismatch between what the model knows how to predict at each timestep and what it's input at the next timestep. This paper describes a method of training output sequence models in a way that reduces this mismatch.

Equilibrated adaptive learning rates for non-convex optimization

Karl: An adaptive learning-rate scheme that takes curvature information into account

A Recurrent Latent Variable Model for Sequential Data

Karl: The authors introduce Variational Recurrent Neural Nets, which explicitly model dependencies between latent random variables across timesteps. Think, like, a Kalman filter, but with learned nonlinear dynamics between the hidden states.

Deep Residual Learning for Image Recognition

Dinesh: The 152-layer network from MSRA that won (most categories of) the Imagenet challenge in 2015. They introduce a neat trick to training such deep networks, without running into vanishing gradient issues.

The Loss Surfaces of Multilayer Networks

Edward: It is shown (under certain assumptions) that local minima do not pose an issue to deep networks because the chance of finding a bad local minima decreases exponentially with the depth of the network. I don't pretend to understand the math here, but I think the result is important.

Adaptive Computation Time for Recurrent Neural Networks

Karl: Alex Graves introduces a method of allowing an RNN to dynamically learn to perform a differing number of Iterations, based on the input (instead of performing the exact same structured computation at each input).

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

SV: Folks from DeepMind do image interpretation as inference in a generative model. Given an image you want to identify what are the objects, where they are, relative positions etc. But take a generative approach to this (like DPMs - deformable parts model - but generative.) They use RNNs, attend to one object at a time and learn to use "appropriate number" of inference steps.

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

SR: Suggest using the same dropout noise through all recurrent layers of the mask. They actually justify it theoretically. Emperically it works

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

SR: Techniques for compressing neural networks, including pruning, quantization, and huffman coding. Accuracy perserved, two orders smaller model, ~5x faster.

Sparse Word Embeddings Using L1 Regularized Online Learning

SR: They add L1 regularization to w2v, get more interpretable dimensions. probably not worth reading, but it's nice