From 56ed911edc9a5a3625b2c75bd5a5602a24dfe30d Mon Sep 17 00:00:00 2001 From: Richard Johansson Date: Fri, 28 Nov 2025 14:54:23 +0100 Subject: [PATCH] typos --- _pages/dat450/assignment4.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/_pages/dat450/assignment4.md b/_pages/dat450/assignment4.md index 74d35f41de7d1..a9d102f5f5d2d 100644 --- a/_pages/dat450/assignment4.md +++ b/_pages/dat450/assignment4.md @@ -539,14 +539,17 @@ You can use nn.init.normal_ and nn.init.zeros_ here. ### Fine-tuning with LoRA -Set up a model where you replace the query and value linear layers with LoRA layers. Use the following steps: +Set up a model where you replace the four linear layers in attention blocks (query, key, value, and output) with LoRA layers. Use the following steps: - First use `extract_lora_targets` to get the relevant linear layers. - Each of the linear layers in the returned dictionary should be wrapped inside a LoRA layer. - Then use `replace_layers` to put them back into the model. -**Sanity check**: Use your function `num_trainable_parameters`. The number of trainable parameters should be less than in Step 1 but more than in Step 2. The exact number will depend on the rank. +**Sanity check**: Use your function `num_trainable_parameters`. The number of trainable parameters should be less than in Step 3. The exact number will depend on the rank. + +Train this model and compare the training speed, metrics, and outputs to the results from Step 3. + +**Correction (Nov. 28)**: We fixed a couple of typos here, in particular a mistake in the instructions about which layers you should apply LoRA to. -Train this model and compare the training speed, metrics, and outputs to the results from Steps 1 and 2. **Side notes:** -----