That's right, the randomly initialized weights in the classification head were untouched.

Hugging Face has tutorial on how to fine-tune a model without LoRA. However, to my knowledge this re-trains all the weight, so an extra step would need to be done to freeze the existing model weights. This isn't something I've done before, but my sense is it should be straightforward.

After that steps is done, one could return to the code provided here for additional fine-tuning.

I plan to do another (more practical) fine-tuning tutorial but using a larger model like Llama-2 7b, Mixtral 7b, or Phi-2.