# Luong Attention

Luong Attention is proposed in the paper [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025) with two variants

* Global Attention
    
* Local Attention
    

The proposed NMT model architecture consists of LSTM encoder to encode input sequences and LSTM decoder to predict the translation.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1732971266932/5ce99c0a-f475-438d-9f28-e4c676ec34dd.png align="center")

At each time step t in the decoding phase, given the target hidden state \\(h_t\\) and the source-side context vector \\(c_t\\), attentional hidden state is computed as,

$$\mathbf{\tilde{h}}_t = \tanh(\mathbf{W}_c[\mathbf{c}_t;\mathbf{h}_t])$$

and then tokens are predicted as,

$$p(y_t|y_{<t}, x) = Softmax(\mathbf{W}_s\mathbf{\tilde{h}}_t)$$

The difference in global and local attention mechanisms is in the manner source-side context vector \\(\mathbf{c}_t\\) is computed.

## Global Attention

Global attentional model considers all the hidden states of the encoder when deriving the context vector \\(c_t\\).

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733080464702/505b0475-ed7a-484a-b83d-9d93273eb1a2.png align="center")

* For the current target hidden state \\(h_t\\) and each source hidden state \\(\bar{h}_s\\), attention score is computed as,
    

$$\begin{align} a_t(s) &= align(h_t, \bar{h}_s) \\ &= \frac{exp(score(h_t, \bar{h}_s))}{\sum_{s'}exp(score(h_t, \bar{h}_{s'}))} \end{align}$$

* Authors propose different methods to compute score,
    

$$\begin{equation} score(h_t, \bar{h}_s) = \begin{cases} h_t^T\bar{h}_s & dot\\ h_{t}^{T}W_a\bar{h}_s & general\\ v_{a}^{T}tanh(W_a[h_t;\bar{h}_s]) & concat \end{cases} \end{equation}$$

* Global attention vector is then computed as weighted average over all the source states
    

$$c_t = \sum_s{a_t(s)\bar{h}_s}$$

## Local Attention

Local attentional mechanism chooses to focus only on a small subset of the source positions per target word.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733082573859/0d01ef72-0226-4cbd-84ad-751a0e16fd58.png align="center")

* For each decoding step t, an aligned position \\(p_t\\) in source sequence can estimated by the following methods,
    
    * Monotonic alignment (local-m): simply set \\(p_t = t\\) assuming that source and target sequences are roughly monotonically aligned
        
    * Predictive alignment (local-p): for given input sequence length S, predict an aligned position as,
        

$$p_t = S \cdot sigmoid(v_p^⊤tanh(W_ph_t))$$

* Context vector \\(c_t\\) is then derived as weighted average of encoder hidden states in the window \\([p_t-D, p_t+D]\\), where D is empirically estimated
    
* Further, to favor alignment points near \\(p_t\\), a Gaussian distribution is placed centered around \\(p_t\\). As a result, the alignment weights are defined as,
    

$$\begin{align} a_t(s) &= align(h_t, \bar{h}_s) exp\left(-\frac{(s-p_t)^2}{2\sigma^2}\right) \\ \sigma &= \frac{D}{2} \end{align}$$

Where s is an integer within the context window centered at \\(p_t\\) and \\(align(h_t, \bar{h}_s)\\) is same as defined for Global Attention.