Daily Archives: Thursday, 16 June 2016

Attention and Memory in Deep Learning and NLP

A good introductory overview of attention models in neural networks.

Attention Mechanisms in Neural Networks are (very) loosely based on the visual attention mechanism found in humans. Human visual attention is well-studied and while there exist different models, all of them essentially come down to being able to focus on a certain region of an image with “high resolution” while perceiving the surrounding image in “low resolution”, and then adjusting the focal point over time.

Attention in Neural Networks has a long history, particularly in image recognition. […] But only recently have attention mechanisms made their way into recurrent neural networks architectures that are typically used in NLP (and increasingly also in vision). That’s what we’ll focus on in this post.

Full article: Attention and Memory in Deep Learning and NLP – WildML

Tagged ,