Attention and Memory in Deep Learning and NLP

A good introductory overview of attention models in neural networks.

Attention Mechanisms in Neural Networks are (very) loosely based on the visual attention mechanism found in humans. Human visual attention is well-studied and while there exist different models, all of them essentially come down to being able to focus on a certain region of an image with “high resolution” while perceiving the surrounding image in “low resolution”, and then adjusting the focal point over time.

Attention in Neural Networks has a long history, particularly in image recognition. […] But only recently have attention mechanisms made their way into recurrent neural networks architectures that are typically used in NLP (and increasingly also in vision). That’s what we’ll focus on in this post.

Full article: Attention and Memory in Deep Learning and NLP – WildML

Tagged ,

Why Hollywood ignores Hiroshima

Ronald Bergan ponders why American films are so reluctant to depict the Hiroshima bombing.

In the opening dialogue of Alain Resnais’s masterful Hiroshima Mon Amour (1959), the reference to which all other films on the subject must incline, a French actor in Hiroshima for a film, tells her Japanese lover that she has seen everything in Hiroshima – the exhibits in the museum, the news footage of the injured and dying. However, he keeps insisting, “You saw nothing in Hiroshima. Nothing.”

American cinema has seen nothing in Hiroshima. Nor has it ever tried.

Read the full article on The Guardian.

Tagged ,

Howdoi – Instant Coding Answers via Command Line

The idea is simple – ask a question and get an answer.

$ howdoi format date bash
> DATE=`date +%Y-%m-%d`

Howdoi solves life’s little coding mysteries like the always frustrating command line flags for tar (as illustrated by XKCD).

Github repo
Blog post describing the tool.

Show and Tell: A Neural Image Caption Generator

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU score the higher the better on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU score improvements on Flickr30k, from 55 to 66, and on SBU, from 19 to 27.

Full article and PDF.

via Google Research Blog

Tagged , , ,

DeepLearning.University: an annotated deep learning bibliography

DeepLearning.University – An Annotated Deep Learning Bibliography | Memkite.

An impressive, annotated bibliography of recent deep learning papers. Doesn’t include publications before 2014 but still quite impressive and comprehensive.

Tagged ,

Digital X-ray images in 5 seconds

Wanting to replace the medical equipment for taking X-rays, experts in Mexico have created a system of digital x-ray imaging, which replaces the traditional plaque by a solid detector, which delivers results in five seconds. Analog equipment take six minutes to develop the traditional film.

Read the full article on ScienceDaily.


Machine Learning, meet Computer Vision

Computer vision, the field of building computer algorithms to automatically understand the contents of images, grew out of AI and cognitive neuroscience around the 1960s. “Solving” vision was famously set as a summer project at MIT in 1966, but it quickly became apparent that it might take a little longer! The general image understanding task remains elusive 50 years later, but the field is thriving. Dramatic progress has been made, and vision algorithms have started to reach a broad audience, with particular commercial successes including interactive segmentation available as the “Remove Background” feature in Microsoft Office, image search, face detection and alignment, and human motion capture for Kinect. Almost certainly the main reason for this recent surge of progress has been the rapid uptake of machine learning ML over the last 15 or 20 years.

This first post in a two-part series will explore some of the challenges of computer vision and touch on the powerful ML technique of decision forests for pixel-wise classification.

Read the full article by Jamie Shotton, Antonio Criminisi and Sebastian Nowozin on TechNet Blogs – Machine Learning.

Tagged , , ,

A Brief History of Machine Learning

Since the initial standpoint of science, technology and AI, scientists following Blaise Pascal and Von Leibniz ponder about a machine that is intellectually capable as much as humans. Famous writers like Jules Verne , Frank Baum (Wizard of OZ), Marry Shelly (Frankenstein), George Lucas (Star Wars) dreamed artificial beings resembling human behaviors or even more, swamp humanized skills in different contexts.

Machine Learning is one of the important lanes of AI which is very spicy hot subject in the research or industry. Companies, universities devote many resources to advance their knowledge. Recent advances in the field propel very solid results for different tasks, comparable to human performance (98.98% at Traffic Signs – higher than human-).

Here I would like to share a crude timeline of Machine Learning and sign some of the milestones by no means complete. In addition, you should add “up to my knowledge” to beginning of any argument in the text.

Read the full article on Eren Golge’s Blog.


Power to us, the people – Sharing The Nation by Zainah Anwar

Let me share this statement, as food for thought, which has been attributed to German theologian and social activist Martin Niemoller who criticised German intellectuals for keeping quiet while the Nazis purged one group of people after another:

“First they came for the Jews, and I did not speak out; Because I was not a Jew. Then they came for the Socialists, and I did not speak out; Because I was not a Socialist.

“Then they came for the Trade Unionists, and I did not speak out; Because I was not a Trade Unionist.

“Then they came for me; And there was no one left to speak for me.”

So if our political leaders have neither the will nor the courage to do what is needed and what is right for this nation, then we the people must show them. We must show them we want to build bridges, live together, understand and respect each other. This is as much our country as it is theirs.

Read the full article on The Star Online.

Tagged , ,

Googles Game Of Moneyball In The Age Of Artificial Intelligence

Over the past couple of months, Google has been playing its own peculiar game of Moneyball. It may not make a ton of sense right now, but Google is setting itself up to leave its competitors in the lurch as it moves into the next generation of computing.

Read the full article on ReadWrite.

Tagged , , ,