Luong Attention

Luong Attention is proposed in the paper Effective Approaches to Attention-based Neural Machine Translation with two variants

Global Attention
Local Attention

The proposed NMT model architecture consists of LSTM encoder to encode input sequences and LSTM decoder to predict the translation.

At each time step t in the decoding phase, given the target hidden state $h_t$ and the source-side context vector $c_t$, attentional hidden state is computed as,

$$\mathbf{\tilde{h}}_t = \tanh(\mathbf{W}_c[\mathbf{c}_t;\mathbf{h}_t])$$

and then tokens are predicted as,

$$p(y_t|y_{

The difference in global and local attention mechanisms is in the manner source-side context vector $\mathbf{c}_t$ is computed.

Global Attention

Global attentional model considers all the hidden states of the encoder when deriving the context vector $c_t$.

For the current target hidden state $h_t$ and each source hidden state $\bar{h}_s$, attention score is computed as,

$$\begin{align} a_t(s) &= align(h_t, \bar{h}_s) \\ &= \frac{exp(score(h_t, \bar{h}_s))}{\sum_{s'}exp(score(h_t, \bar{h}_{s'}))} \end{align}$$

Authors propose different methods to compute score,

$$\begin{equation} score(h_t, \bar{h}_s) = \begin{cases} h_t^T\bar{h}_s & dot\\ h_{t}^{T}W_a\bar{h}_s & general\\ v_{a}^{T}tanh(W_a[h_t;\bar{h}_s]) & concat \end{cases} \end{equation}$$

Global attention vector is then computed as weighted average over all the source states

$$c_t = \sum_s{a_t(s)\bar{h}_s}$$

Local Attention

Local attentional mechanism chooses to focus only on a small subset of the source positions per target word.

For each decoding step t, an aligned position $p_t$ in source sequence can estimated by the following methods,
- Monotonic alignment (local-m): simply set $p_t = t$ assuming that source and target sequences are roughly monotonically aligned
- Predictive alignment (local-p): for given input sequence length S, predict an aligned position as,

$$p_t = S \cdot sigmoid(v_p^⊤tanh(W_ph_t))$$

Context vector $c_t$ is then derived as weighted average of encoder hidden states in the window $[p_t-D, p_t+D]$, where D is empirically estimated
Further, to favor alignment points near $p_t$, a Gaussian distribution is placed centered around $p_t$. As a result, the alignment weights are defined as,

$$\begin{align} a_t(s) &= align(h_t, \bar{h}_s) exp\left(-\frac{(s-p_t)^2}{2\sigma^2}\right) \\ \sigma &= \frac{D}{2} \end{align}$$

Where s is an integer within the context window centered at $p_t$ and $align(h_t, \bar{h}_s)$ is same as defined for Global Attention.

Luong Attention

Comments

Attention Mechanism

Bahdanau Attention

More from this blog

DIN: Deep Interest Network for Click-Through Rate Prediction

AttriBERT - Session-based Product Attribute Recommendation with BERT

Interpretable Attribute-based Action-aware Bandits for Within-Session Personalization in E-commerce

Hard, Soft and Doubly Stochastic Attention

Global Attention

Local Attention

Command Palette

Comments

Attention Mechanism

Bahdanau Attention

More from this blog

Global Attention

Local Attention