Thanks Pranav, great questions.

1) I wanted the example to be simple (and binary classification is as simple as it gets). But I also wanted to demonstrate how to use LoRA, since it's practically how many people will be able to fine-tune models.

If this was a real-world use case, I'd probably try transfer learning first then fine-tune further with LoRA if necessary.

2) No the head was frozen which may explain the overfitting here 😅

Thanks for raising these points, it's super helpful to me and to other readers as well.