What do you mean by Attention?

Question: 

What do you mean by Attention? What are the types of Attention in Neural Networks?

Expected Answer:

Define attention as a representation of a distribution learnt by a Neural Network, use case and an example. You could also specify types of attention and give applications of each.

Neural attention mechanism equips a neural network with the ability to focus on a subset of its inputs (or features): it selects specific inputs. Let x∈d$\mathbf{x}\in {\mathcal{R}}^d$ be an input vector, z∈k$\mathbf{z}\in {\mathcal{R}}^k$ a feature vector, a∈[0,1]k$\mathbf{a}\in [0,1{]}^k$ an attention vector, g∈k$\mathbf{g}\in {\mathcal{R}}^k$ an attention glimpse and fϕ(x)$f_{\varphi }(\mathbf{x})$ an attention network with parameters ϕ$\varphi$. Typically, attention is implemented as

ag=fϕ(x),=a⊙z,(1)$$$$

where ⊙ is element-wise multiplication, while z is an output of another neural network fθ(x) with parameters θ. We can talk about soft attention, which multiplies features with a (soft) mask of values between zero and one, or hard attention, when those values are constrained to be exactly zero or one, namely a∈{0,1}k. In the latter case, we can use the hard attention mask to directly index the feature vector: g̃ =z[a] , which changes its dimensionality and now g̃ ∈mwith m≤k.

Types of Attention:

1. Visual

2. Hard Attention

3. Soft Attention

4. Gaussian Attention

Read more:

http://akosiorek.github.io/ml/2017/10/14/visual-attention.html