LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen submitted 'LoRA: Low-Rank Adaptation of Large Language Models' to arXiv on June 17, 2021. The paper proposes freezing pretrained model weights while injecting trainable low-rank decomposition matrices into Transformer layers to reduce the number of trainable parameters needed for downstream adaptation.