LoRA (low-rank adaptation)
Also known as: LoRA, low-rank adaptation, LoRA adapter, LoRA fine-tuning
LoRA, short for low-rank adaptation, is a technique for customizing an AI model by training a small set of extra weights instead of changing the whole model.
The add-on, often called an adapter, can be a tiny fraction of the model's size and can be swapped in or out. While LoRA has made fine-tuning affordable for small teams and hobbyists, it can't add much new knowledge, and a poorly trained adapter can make a model worse.
LoRA (low-rank adaptation), explained at three levels
At home
Think of an AI model as a huge, finished textbook. Rewriting the whole book for every class would be expensive, so instead you add a thin stack of sticky notes that adjust certain answers. LoRA is that stack of notes. The original model stays untouched, and the small add-on, often just megabytes, changes its style or behaviour and can be shared or swapped out.
At work
LoRA, short for low-rank adaptation, is a cheap way to customize an AI model. Instead of retraining billions of parameters, it trains a small add-on called an adapter. Businesses use it to teach a model their tone, document format, or a narrow task at a fraction of the hardware cost, and can keep separate adapters for different teams on one base model.
At home
A LoRA file is a set of small weight adjustments trained on a few dozen to a few thousand examples, such as 20 pictures of your dog or a set of paintings in one style. You load the base model, such as an open image generator, then load the LoRA on top. You can usually set its strength, and even stack several, like one for a character and one for an art style.
At work
In a neural network, each layer multiplies its inputs by a large weight matrix. LoRA freezes those matrices and learns the change as two thin matrices whose product approximates the update; the 'rank' sets how thin they are, often 8 to 64. That can cut trainable parameters by a factor of thousands. Adapters can be merged into the base weights for speed or kept separate, letting one base model serve many customers. Key settings are rank, learning rate, and target layers.
At home
Because the update is low-rank, LoRA can only nudge the model in a limited number of directions. That suits style, tone, and format, which are fairly simple shifts, but storing many new facts needs far more capacity than an adapter has. While you can train a LoRA on your documents, the model will still guess about details. In my opinion, if you want an AI to know your documents, retrieval usually works better.
At work
LoRA has trade-offs. Research shows it learns less than full fine-tuning in hard domains like code and math, though it also forgets less of the base model's skills. Variants address its limits: QLoRA trains on a 4-bit quantized base to save memory, and DoRA splits weights into magnitude and direction for better quality. Serving engines such as vLLM can host many adapters on one base model. I'd still evaluate against the base model and check its licence terms.
Real-world examples
At home
Sophie trained a small LoRA on her own watercolour paintings. The image generator could then produce new pictures in her style for her family holiday cards.
At work
Carlos is an AI engineer at an Ottawa software firm. He used LoRA to fine-tune an open model to write support replies in the company's format, training it in an afternoon on one rented GPU.
How it compares
| feature | LoRA (low-rank adaptation) | Full fine-tuning |
|---|---|---|
| What's trained | Small add-on weights | All model weights |
| Hardware needed | Modest | Large |
| Storage per version | Megabytes | Full model copy |
| Quality | Close to full on many tasks | Highest ceiling |
LoRA gets most of the benefit of fine-tuning at a fraction of the cost.
For analysts
Before fine-tuning, I'd confirm the problem isn't better solved by prompting or retrieval. If fine-tuning is right, LoRA is usually the sensible first step.
History & origin
Fine-tuning large models originally meant updating every parameter, which required expensive hardware and stored a full copy for each custom version. Edward Hu and colleagues at Microsoft published LoRA in 2021, showing that small 'low-rank' updates could match full fine-tuning on many tasks. The technique spread quickly to image models, where hobbyists shared LoRA files to teach Stable Diffusion new styles or characters. In May 2023, Tim Dettmers and colleagues at the University of Washington introduced QLoRA, combining LoRA with quantization so large models could be fine-tuned on a single GPU. Cloud AI platforms now offer LoRA-based fine-tuning as a standard service. In my view, LoRA did more to democratize AI customization than any other single paper.