LoRA vs Full Fine-Tuning: What Do We Actually Save?

中文导读

LoRA vs full fine-tuning 这个问题不能只问“哪个分数更高”。更实际的问题是：在同样的数据、算力和调参预算下，哪个方法更值得用？

在我的 English—Norwegian petroleum MT 项目里，LoRA 用极少的可训练参数接近 full fine-tuning 的性能，并且在低资源设置下更稳定。这篇文章整理这个 trade-off。

Two adaptation paradigms

Full fine-tuning updates all model parameters. It gives the model maximum flexibility.

LoRA freezes the base model and learns small low-rank updates in selected modules. It gives the model controlled flexibility.

The difference is not just technical. It affects cost, stability, reproducibility, and how many experiments we can afford to run.

What full fine-tuning gives us

Full fine-tuning is powerful because it can modify the entire model. If the dataset is large and clean, and compute is available, this flexibility can be valuable.

Potential advantages:

maximum adaptation capacity;
no low-rank bottleneck;
potentially better performance with enough data;
direct optimization of all model parameters.

But these advantages come with costs:

higher memory usage;
more trainable parameters;
greater storage cost for checkpoints;
higher overfitting risk in low-resource settings;
more expensive hyperparameter search.

What LoRA gives us

LoRA gives us a smaller adaptation mechanism.

Potential advantages:

far fewer trainable parameters;
lower memory requirements;
easier repeated experiments;
smaller adapter checkpoints;
stable adaptation with limited data;
possible merging into base weights for inference.

In my project, LoRA updated less than one percent of the model parameters while reaching competitive translation quality.

This is why LoRA is attractive for academic and applied settings where GPU resources are real constraints.

What do we actually save?

When people say LoRA is efficient, they often mean parameter-efficient. But in practice, it saves several things.

1. Trainable parameters

Only adapter parameters are updated. This reduces the number of gradients and optimizer states.

2. Memory

Because fewer parameters require gradients and optimizer states, memory usage can be lower.

3. Storage

Adapters are much smaller than full model checkpoints. This makes it easier to keep multiple experimental variants.

4. Tuning cost

If LoRA is stable across data scales, a tuned configuration can be reused as a starting point for similar tasks.

5. Experimental agility

When each run is cheaper, it becomes easier to test data sizes, random seeds, cleaning strategies, and evaluation variants.

This last point matters a lot in research. Efficiency is not only about deployment; it also changes what experiments are feasible.

Does LoRA lose performance?

Sometimes, yes.

LoRA constrains adaptation to a low-rank update. If the task requires broad changes across the model, full fine-tuning may perform better.

But in low-resource domain MT, the gap may be small. The pretrained model already knows general translation. The adaptation mainly needs to adjust terminology, style, and domain-specific mappings.

In my experiments, full fine-tuning was sometimes slightly better at medium data sizes, but the gap was small and tended to shrink as the task approached saturation.

This suggests that LoRA was not severely limiting the model for this domain.

The fairness question

Comparing LoRA and full fine-tuning is tricky.

If full fine-tuning receives a large dedicated hyperparameter search, it may improve. But that also increases compute cost.

A practical comparison asks:

Given equal tuning effort, which method is more useful?

Under that framing, LoRA is often attractive because it approaches full fine-tuning performance without requiring the same level of compute investment.

When I would choose LoRA

I would choose LoRA when:

the dataset is small or medium-sized;
the base model is already strong;
the goal is domain adaptation;
compute is limited;
many experiments are needed;
checkpoint storage matters;
I want to compare multiple data or prompt conditions.

When I would consider full fine-tuning

I would consider full fine-tuning when:

there is enough high-quality data;
domain shift is very large;
maximum performance matters more than cost;
compute resources are sufficient;
the model needs deeper behavioral change.

Takeaway

LoRA does not only save parameters. It saves experimental budget.

For low-resource domain MT, that can be the difference between running one model and running a systematic study with data scaling, hyperparameter optimization, multiple seeds, and richer evaluation.

That is why I see LoRA not only as an efficient adaptation method, but as a research-enabling method.