Transformers and Attention 🔮

Before transformers, text was processed by Recurrent Neural Networks (RNNs) and LSTMs. They process tokens one at a time, left-to-right, carrying a "hidden state" forward.

• Vanishing gradient: Gradients shrink as they flow back through many time steps. The model forgets early context when processing long sequences.
• No parallelisation: Each token must wait for the previous one. You can't use GPU parallelism — training is slow.
• Long-range dependency failure: In the sentence "The musical group that won the prize after competing for ten years were happy" — "group" and "were" are far apart. RNNs often get the agreement wrong.

The 2017 paper "Attention Is All You Need" (Vaswani et al.) introduced the Transformer, which fixed all three problems at once.

Attention lets each token "look at" every other token in the sequence simultaneously and decide how much to weight each one. Imagine reading the sentence: "The apple trees in the valley it grew in were beautiful."

When processing "it", attention asks: which word does "it" refer to? The model looks at all previous words, assigns high weight to "valley" and "apple trees", and low weight to "grew" and "were".

Self-attention uses three learned matrices:

• Query (Q): "What am I looking for?" — represents the current token asking a question.
• Key (K): "What do I have to offer?" — represents each other token's "identity".
• Value (V): "What information do I carry?" — the actual content to aggregate if attended to.

Below is a simplified attention weight heatmap for the sentence "The river in Srinagar flows swiftly":

Read each row as: "When processing this word, how much attention is paid to each other word?" Darker green = higher attention. Notice "flows" attends strongly to "river" — it knows what is flowing.

Multi-head attention: Run attention multiple times in parallel with different Q, K, V weight matrices. Each "head" learns to attend to different kinds of relationships. For example, one head might track subject-verb agreement, another tracks coreference (pronoun → noun).

Positional encoding: Since transformers process all tokens in parallel, there's no inherent sense of order. Positional encodings are added to token embeddings to inject position information using sine and cosine functions at different frequencies.

# Visualising Attention with HuggingFace Transformers — Google Colab
!pip install transformers bertviz -q

from transformers import BertTokenizer, BertModel
from bertviz import head_view
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model     = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)

sentence = "The river in Srinagar flows swiftly through the valley"
inputs   = tokenizer(sentence, return_tensors='pt')

with torch.no_grad():
    outputs = model(**inputs)

attention = outputs.attentions      # tuple of (layer, batch, heads, seq, seq)
tokens    = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])

print(f"Layers: {len(attention)}")  # 12 layers in BERT-base
print(f"Heads per layer: {attention[0].shape[1]}")   # 12 heads
print(f"Sequence length: {attention[0].shape[-1]}")  # tokens

# Interactive visualisation — run in Colab to see coloured attention arcs
head_view(attention, tokens)