All projects

Project 3 of 3

Team project

Fine-tuning BERT and DistilBERT with LoRA

Four automated training runs on the Stanford Natural Language Inference dataset, to compare fine-tuning methods.

DistilBERT test accuracy

  • LoRA 83.48%
  • Partial fine-tuning 81.12%
Illustration of a large grid of frozen weights plus two small trained grids, the idea behind low-rank adaptation.
Period
Nov 2025 – Jan 2026
Type
Team project
Course
Advanced Machine Learning for Natural Language Processing
University
IT University of Copenhagen
Models
  • BERT
  • DistilBERT
Methods
Low-rank adaptation (LoRA), partial fine-tuning
Dataset
Stanford Natural Language Inference
Code
GitHub repository

Method

  1. Automated four training runs on the Stanford Natural Language Inference dataset
  2. Logged accuracy, training time and peak GPU memory
  3. Compared low-rank adaptation with partial fine-tuning