VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models
In the authors' words
Fine-tuning public pretrained language models with lightweight adapters such as LoRA has become the mainstream way to adapt them to downstream tasks. A service provider often trains its adapter on a data owner's samples, so two assets need protection: the data owner's samples and the provider's adapter. Existing secure fine-tuning protocols protect the samples but reveal to the data owner intermediate results of every training step, such as the hidden representations or logits of its inputs. We show that a single step suffices to copy the adapter: from the hidden representations of one mini-batch of 32 samples, the data owner fits an adapter that agrees with the provider's on 99.2% of test inputs, and logits alone yield about 94%. Protecting both assets therefore requires that no value that depends on the adapter be revealed throughout training. We present VeilTune, to the best of our knowledge the first protocol that fine-tunes a pretrained language model with backpropagation entirely in secure multi-party computation (MPC). VeilTune computes only what the loss depends on, replaces generic MPC operators with training-aware nonlinear operators that stay accurate on the values that training produces, and obliviously skips the steps corrupted by truncation failures. On four few-shot tasks with RoBERTa-large, VeilTune nearly matches plaintext training and is 5.3 points more accurate on average than SecP-Tuning, the state-of-the-art secure fine-tuning protocol. On SST-2, it achieves a 3.5x end-to-end speedup and a 3.6x reduction in communication over SecP-Tuning, and 26x and 17x over full fine-tuning in MPC.
In plain words AI summary
Fine-tuning public language models with lightweight adapters like LoRA is now common. Current secure methods protect data samples but reveal intermediate training results to data owners. The authors show one training step can copy the adapter: hidden representations from 32 samples yield 99.2% agreement, logits about 94%. They present VeilTune, the first protocol to fine-tune with backpropagation entirely in secure multi-party computation. VeilTune nearly matches plaintext training and beats the prior secure method by 5.3 points on average, with 3.5x speedup on SST-2.
Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.
Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.
Authors' comment: 13 pages, 7 tables