Week 2: Attention & Transformers

Introduction

Transformers are the foundational architecture behind most modern large language models, including BERT, GPT, and T5. Introduced in the groundbreaking paper “Attention Is All You Need”¹, transformers leverage self-attention mechanisms to model dependencies across sequences, enabling scalable and parallel training. In this week, we will explore how transformers process input tokens, how attention mechanisms function internally, and how architectures evolve when scaling to billions of parameters. This foundational knowledge is critical for understanding how LLMs generate, reason, and attend to context.

Goals for the Week:

• Understand the core components of the Transformer architecture: embeddings, multi-head self-attention, positional encoding, feed-forward layers, and layer normalization.
• Trace the flow of information through a Transformer block.
• Analyze architectural variants like, BERT² and GPT³ to see how they adapt the core design.
• Get familiar with HuggingFace transformers library.

Learning Guide

• Beginner Friendly
  • What is Attention in Language Models?
  • What are Transformer Models & How Do They Work
  • 3Blue1Brown - Transformers, the tech behind LLMs
• Essential Readings
  • The Illustrated Transformer
• Going Deeper
  • Transformers, what can they do?
  • How do Transformers work?
  • How Transformers solve tasks
  • Transformer Architectures

Programming Practice

• Use HuggingFace transformer pipeline to demonstrate and solve at-least two of the following tasks. Use a dataset of your choice, and submit your work in a notebook form. Include evaluation metrics to measure the performance of your tasks.
  • Image Classification
  • Speech Recognition
  • Text Classification
  • Text Generation
  • Translation
  • Any other use-case of your choice.

Examples

from transformers import pipeline
transcriber = pipeline(
    task="automatic-speech-recognition", model="openai/whisper-base.en"
)
transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
# Output: {'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}

• Notebook: Using Transformer Pipeline

References