MobileSpec
Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates
Your phone trains its own specialists overnight, with forward passes alone.
University of California, Santa Barbara
*Equal contribution
Overview
One generalist on the phone, a specialist for every task.
A phone can only hold an LLM of a few billion parameters, well below frontier models. But the tasks people actually run on their phone are narrow and personal. MobileSpec keeps one shared INT4 model on the device, the generalist, and trains a lightweight specialist for each task the user repeats, on the phone, while it is idle.
Backpropagation does not fit in a phone app's memory, and users rarely label data. So MobileSpec trains with forward passes alone, learning from outcomes the device can check automatically or from the user's own records. Each specialist is a 1–5 MB file applied on top of the untouched INT4 weights.
Specialists of 1–5 MB lift a phone-use agent from 6–23% to 89–100% success on three of five tasks, and MobileSpec trains a personalized 8B model on an iPhone 16 Pro overnight, peaking at 3.5 GB of memory.
The paper’s static figure
Method
FuRA-S: search only the block-spectral coordinates.
Forward-only training builds every update from random perturbations, so it learns quickly only when those perturbations look like the weight changes the task needs. FuRA-S picks a search space that is tiny enough to store per task and shaped like the changes RLVR makes.
Block-spectral coordinates
Split each weight matrix into column blocks and factor every block by an untruncated SVD, \(W_k = L_k\,\mathrm{diag}(s_k)\,R_k^\top\). Only the block singular values \(s_k\) are trained, about 0.03% of the parameters.
Forward-only ES
Score antithetic pairs \(\theta \pm Dz_j\), with each singular value perturbed in proportion to its own size, standardize their fitness, and move toward the better members. Directions are regenerated from seeds and never stored.
Runs on INT4 as is
Because nothing is truncated, the update is exactly a transformation of the layer input. The released INT4 weights and kernels stay untouched, and a specialist stores only its vector \(\delta\).
With \(s'_k = s_k \odot (1+\delta_k)\) and \(R = \mathrm{blockdiag}_k(R_k)\). The frozen \(R\) is stored in 4 bits and shared by all specialists; \(\delta = 0\) recovers the quantized model exactly.
Why it works
The directions of a search space matter more than its size.
ES learns only while some perturbations still beat the current model. In LoRA and full-parameter search spaces that fraction drops to zero within 50 iterations. FuRA-S keeps finding improving perturbations through iteration 125 and reaches the highest accuracy, with as few values as the smaller Uni-LoRA, which drifts furthest from the base model.
Spectral inheritance. Perturbing only the block singular values preserves the spectrum of each weight matrix but rotates both of its frames, and the trained weights inherit the same structure: the rotated frames alone retain over 95% of the accuracy gain on MATH and on Countdown.
Phone agents
From rarely to reliably.
MAI-UI-2B operates real Android apps from screenshots in MobileWorld. On five tasks where it succeeds less than a quarter of the time, a specialist that changes only 516,096 block singular values (0.024% of the model) trains for 15 iterations, 960 episodes. A step-wise fitness credits each action against what the other members did on the same screen, so a good action is rewarded even when the member's own episode fails for other reasons.
RLVR
Full-parameter ES accuracy from 0.05% of the parameters.
On Countdown and MATH, FuRA-S is on par with full-parameter ES while training at most 0.05% of the parameters it perturbs. It outperforms ES over LoRA and VeRA adapters on every model, and without computing a single gradient it surpasses PPO, GRPO and Dr. GRPO on four of five models.
On device
An 8B specialist, trained on an iPhone overnight.
On an iPhone 16 Pro, MobileSpec trains a specialist for one LaMP movie-tagging user on top of the released INT4 Llama-3.1-8B-Instruct, using the likelihood of the user's own records as fitness. The 5.5 GB model is larger than the 5.3 GB an app may use, so MobileSpec streams the decoder layers from flash.
FAQ
Questions people ask
What does MobileSpec do?
It keeps one shared, quantized LLM on the phone (the generalist) and trains a small specialist for each task the user repeats, on the phone itself while it is idle. Training uses forward passes only and learns from signals that arise during use: automatic checks of a task's outcome, or the likelihood of the user's own records.
Why not train with backpropagation or LoRA on the phone?
Backpropagation needs several times the memory of inference and is poorly supported by the quantized dataflow of on-device engines. Forward-only methods avoid that cost, but full-parameter evolution strategies keep a full-precision copy of the weights and make every specialist as large as the model, and LoRA's generic low-rank perturbations soon stop finding improving directions under ES.
What are block-spectral coordinates?
Each weight matrix is split into column blocks, and each block is factored by an untruncated SVD, so W_k = L_k diag(s_k) R_k^T reconstructs the weights exactly. FuRA-S trains only the block singular values s_k (about 0.03% of the parameters of Qwen2.5-1.5B) and freezes L_k and R_k.
Why does searching such a tiny space work?
RLVR mostly rotates the singular vectors of pretrained weights while preserving their singular values (spectral inheritance). Rescaling the singular values of each block independently shifts the blocks' relative contributions to the assembled matrix, which rotates its singular vectors while barely changing its singular values. ES therefore keeps finding improving perturbations for longer, and the rotated frames alone retain over 95% of the accuracy gain.
How does it run on an INT4 model?
Because nothing is truncated, the update can be rewritten exactly as W' = W(I + R diag(delta) R^T), a transformation of the layer input. The released INT4 weights, their scales and zero points, and the existing INT4 kernels stay untouched; each specialist stores only its full-precision delta, and delta = 0 recovers the quantized model exactly.
How well does it work?
On five MobileWorld phone-use tasks, specialists of MAI-UI-2B reach 64–100% success from a 6–23% base. On RLVR benchmarks FuRA-S is competitive with full-parameter ES while training at most 0.05% of the parameters, outperforms ES over LoRA adapters, and beats quantized ES on INT4 models by up to 16.25 points. On an iPhone 16 Pro it trains a personalized Llama-3.1-8B specialist in 10.4 hours with 3.5 GB peak memory.
Citation
BibTeX
Abstract
Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1–5 MB that update 0.015–0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6–23% up to 89–100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.
@misc{tan2026mobilespec,
title = {Training Specialist {LLMs} on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates},
author = {Tan, Liyan and Zhao, Yequan and Liu, Ziyue and Su, Yupeng and Zhang, Ruijie and Zhang, Zheng},
year = {2026},
eprint = {2611.04797},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
note = {Liyan Tan and Yequan Zhao contributed equally}
}