More capacity made the model worse

When we started adapting Lor-1 we assumed more capacity was better: the more of the base we let training rewrite, the more of the South African clinical corpus it should absorb. We were wrong, in a way that took a second benchmark to see. This post is the short version of what happened, without the hyperparameters.

The experiment

Same base, same training data, same schedule. The only variable was how much of the base we allowed the adaptation to move: a conservative run, a middling one, and an aggressive one. Each checkpoint was evaluated on LorBench — our internal SA clinical benchmark — plus a held-out set of general medical-knowledge probes whose only job was to catch catastrophic forgetting.

What we found

The aggressive run posted the highest LorBench score mid-training, then collapsed on the parametric probes — shedding general medical knowledge the base already had right. The middling run held its parametric ability better but never caught the conservative run on the target benchmark. The conservative run was the sweet spot: it absorbed the SA corpus without evicting what the base already knew.

Why this makes sense

A low-rank adaptation projects updates through a narrow subspace. The more capacity you give it, the more of the base the optimiser can effectively rewrite — fine when you are training on generic data, expensive when the base already holds correct knowledge you need to keep. For clinical adaptation you want the smallest change that still fits the new distribution.

Operational takeaway

We now treat the conservative setting as the default for clinical adaptation, and only reach for more capacity when the target really is a domain the base has never seen — which, in medicine, is rarely the case. The exact settings sit in the private disclosure alongside the rest of the pipeline, not in a blog post.