Introduction

Fine-tuning adapts pre-trained language models to specific tasks or behaviors. This week focuses on two core strategies: (1) supervised fine-tuning for classification and structured tasks using encoder models like BERT, and (2) instruction fine-tuning of decoder-only LLMs (e.g., LLaMA) to follow human instructions or exhibit dialogue-friendly behavior. We’ll cover how fine-tuning changes model weights to align with task objectives, when to use parameter‑efficient techniques like LoRA¹ or adapters², and how to interact with pre‑fine‑tuned instruct models using chat templates.

Goals for the Week

• Understand the difference between task-specific fine-tuning and instruction tuning.
• Fine-tune a classification model (e.g., BERT) using Hugging Face Transformers and Datasets.
• Experiment with chat-oriented LLMs using Chat Templates without performing full fine-tuning.
• Learn the challenges and infrastructure requirements of full supervised fine-tuning (SFT)³.
• Explore parameter-efficient methods like LoRA¹ and adapters².
• Evaluate the effects of fine-tuning on output quality and behavior.

Learning Guide

Videos

Fine-tuning LLMs: Short Course (1 hr 25 mins)

• Reference material: ksm26/Finetuning-Large-Language-Models

Readings

Task-Specific Supervised Fine-Tuning (classification, seq labeling, etc.)
Fine-tune encoder(-decoder) models (e.g., BERT/T5) on labeled tasks to optimize task-specific losses (e.g., classification cross-entropy).

• Fine-Tuning for Text Classification – Hugging Face
  • Example: fine-tuning a BERT model for sequence classification. With a labeled downstream dataset, set up a training loop to adapt the base model to your task.
• Fine-Tuning for Classification – Cohere
  • Similar objective using the Cohere API.

Instruction Fine-Tuning (SFT for chat/instructions)
Supervised next-token training of decoder-only LLMs on instruction→response or chat data to improve instruction following and dialogue behavior.

• Chat Templates

  • Decoder based generative models (base) are instruction-fine-tuned to work in conversational manner. The process is called Supervised Fine-Tuning is not an easy to do process by oneself. Ready to use instruct models are available on HuggingFace for us to experiment with. Using ChatTemplates we can assign roles to model and prompt it to desired feedback.

• Chat with Models

• Multimodal: Chat Templates for image/video + text

• Cohere API for Instruction Fine-Tuning

• Fine-Tuning with SFTTrainer

  • SFTTrainer

Parameter-Efficient Fine-Tuning (PEFT)

• Techniques include LoRA¹, adapters², prefix-tuning⁴, and P-Tuning v2⁵
• QLoRA⁶ for efficient fine-tuning of quantized large language models
• LoRA: Low-Rank Adaptation
  • Video Explanation

RLHF Pipeline (optional/advanced)

• Instruction tuning with human feedback (RLHF)³: SFT on human demonstrations → reward modeling → PPO

Programming Practice

• Task-Specific Supervised Fine-Tuning: Fine-tune a bert-base-uncased model on the IMDb or AG News classification dataset using Hugging Face Transformers.
• Instruction Fine-Tuning: Load a pre‑fine‑tuned instruction‑following model like HuggingFaceH4/zephyr-7b-beta and interact with it using chat templates and role prompting.
• Multimodal Chat Templates (Optional): Build a short prompt that mixes text plus an image using AutoProcessor.apply_chat_template and run it with an image-text-to-text pipeline.
• Supervised Fine-Tuning (Optional): Write a custom conversation dataset and simulate fine‑tuning using the SFTTrainer (TRL) or peft in dry‑run mode (small batch size, limited steps).
• Visualize the effect of fine-tuning by comparing predictions before and after adaptation.

References