MobileSpec
arXiv 2026

MobileSpec

Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

Your phone trains its own specialists overnight, with forward passes alone.

University of California, Santa Barbara

*Equal contribution

89–100%phone-agent success after training, up from 6–23%
1–5 MBper specialist, 0.015–0.05% of the parameters
3.5 GBpeak memory training an 8B model on an iPhone 16 Pro
+16.25points over quantized ES on an INT4 model

Overview

One generalist on the phone, a specialist for every task.

A phone can only hold an LLM of a few billion parameters, well below frontier models. But the tasks people actually run on their phone are narrow and personal. MobileSpec keeps one shared INT4 model on the device, the generalist, and trains a lightweight specialist for each task the user repeats, on the phone, while it is idle.

Backpropagation does not fit in a phone app's memory, and users rarely label data. So MobileSpec trains with forward passes alone, learning from outcomes the device can check automatically or from the user's own records. Each specialist is a 1–5 MB file applied on top of the untouched INT4 weights.

TL;DR

Specialists of 1–5 MB lift a phone-use agent from 6–23% to 89–100% success on three of five tasks, and MobileSpec trains a personalized 8B model on an iPhone 16 Pro overnight, peaking at 3.5 GB of memory.

A specialist, trained overnight Interactive

Pick a task the phone agent keeps failing, then train its specialist. Perturbed copies of the model try the task in sandboxed apps, and forward-only ES moves the block singular values toward the better ones. Success rates are the paper’s MobileWorld results for MAI-UI-2B; the members’ episodes are drawn schematically.

The paper’s static figure
On-device training of a specialist with MobileSpec
On-device training of a specialist with MobileSpec. When the generalist agent fails at a task the user repeats, MobileSpec trains a specialist for it overnight. Perturbed models attempt the task in sandboxed apps, and forward-only ES moves the block singular values toward those of the better members. The specialist is a 2 MB vector δ applied on top of the shared INT4 generalist.

Method

FuRA-S: search only the block-spectral coordinates.

Forward-only training builds every update from random perturbations, so it learns quickly only when those perturbations look like the weight changes the task needs. FuRA-S picks a search space that is tiny enough to store per task and shaped like the changes RLVR makes.

1

Block-spectral coordinates

Split each weight matrix into column blocks and factor every block by an untruncated SVD, \(W_k = L_k\,\mathrm{diag}(s_k)\,R_k^\top\). Only the block singular values \(s_k\) are trained, about 0.03% of the parameters.

2

Forward-only ES

Score antithetic pairs \(\theta \pm Dz_j\), with each singular value perturbed in proportion to its own size, standardize their fitness, and move toward the better members. Directions are regenerated from seeds and never stored.

3

Runs on INT4 as is

Because nothing is truncated, the update is exactly a transformation of the layer input. The released INT4 weights and kernels stay untouched, and a specialist stores only its vector \(\delta\).

$$W' = W\,(I + E), \qquad E = R\,\mathrm{diag}(\delta)\,R^\top, \qquad y = \underbrace{W_q}_{\text{INT4, frozen}}\bigl(x + R\,\mathrm{diag}(\delta)\,R^\top x\bigr)$$

With \(s'_k = s_k \odot (1+\delta_k)\) and \(R = \mathrm{blockdiag}_k(R_k)\). The frozen \(R\) is stored in 4 bits and shared by all specialists; \(\delta = 0\) recovers the quantized model exactly.

Why rescaling blocks turns the whole matrix Animated

The ellipse is where a toy weight matrix sends the unit sphere: its axes are the output frame, their lengths the singular values. On the right the matrix has two column blocks. Each block keeps its own axes (orange, green) and only stretches, but the two stretch by different amounts, so the axes of the whole matrix, set by \(WW^\top=\sum_k L_k\,\mathrm{diag}(s_k)^2 L_k^\top\), turn. Rescaling the global singular values (left) stretches the axes in place.

whole matrix and its axis u₁block 1block 2before

Rescaling every block by the same factor rescales the whole matrix and turns nothing. The turn comes from rescaling the blocks independently, which is exactly what ES perturbations do.

Rescale the blocks, rotate the frames Live SVD

A 12×12 toy weight matrix with a decaying spectrum. Perturb its block singular values and watch the singular values of the whole matrix barely move while its singular vectors rotate. With one block, FuRA-S collapses to rescaling the global singular values, and the frames stop rotating.

3%

Rotation is the largest principal angle between the leading 3-dimensional singular subspaces before and after the update; spectrum change is the relative change of the singular values. Every update is scaled to the same ‖ΔW‖. Toy numbers; on Qwen2.5-1.5B matrices the paper measures about 9° of rotation, four times that of a dense update.

Why it works

The directions of a search space matter more than its size.

ES learns only while some perturbations still beat the current model. In LoRA and full-parameter search spaces that fraction drops to zero within 50 iterations. FuRA-S keeps finding improving perturbations through iteration 125 and reaches the highest accuracy, with as few values as the smaller Uni-LoRA, which drifts furthest from the base model.

Frames rotate, spectrum stays 3D · drag to rotate

A toy 3×12 weight matrix and a task whose best weights keep its spectrum but turn its output frame by 23°, the kind of change RLVR makes. ES over the 12 block singular values carries the frame to the target ★, with the 16 perturbed members of each iteration in orange. ES over the global singular values can only stretch the axes in place and stalls.

ES iteration0
Output frame turnedangle of \(u_1\) from \(U_0\)0°
Spectrum change0%
Distance to targetrelative to the start100%
blockglobal

On real weights

Rotation of the leading singular vectors of eight Qwen2.5-1.5B-Instruct matrices under updates of the same size as the one FuRA-S learns on Countdown.

The gain lives in the frames Interactive

Split every trained weight matrix into its frames and its spectrum, \(W=U_T\,[(1-\alpha)\Sigma_0+\alpha\Sigma_T]\,V_T^\top\). Slide \(\alpha\) to put the pretrained spectrum back under the trained frames, or switch to the pretrained frames with the trained spectrum. MATH: Qwen2.5-Math-7B, average of MATH500, Minerva and OlympiadBench. Countdown: Qwen2.5-1.5B-Instruct, mean of two seeds.

α = 0

this combinationpretrained modeltrained model

Spectral inheritance. Perturbing only the block singular values preserves the spectrum of each weight matrix but rotates both of its frames, and the trained weights inherit the same structure: the rotated frames alone retain over 95% of the accuracy gain on MATH and on Countdown.

Phone agents

From rarely to reliably.

MAI-UI-2B operates real Android apps from screenshots in MobileWorld. On five tasks where it succeeds less than a quarter of the time, a specialist that changes only 516,096 block singular values (0.024% of the model) trains for 15 iterations, 960 episodes. A step-wise fitness credits each action against what the other members did on the same screen, so a good action is rewarded even when the member's own episode fails for other reasons.

Success rate by task

Each column is a MobileWorld task, evaluated on 64 new episodes. The line rises from the generalist to the specialist trained with step-wise fitness; the orange dot is the same training with episode rewards only. The top row is the gain.

GeneralistEpisode rewardStep-wise fitness (ours)

RLVR

Full-parameter ES accuracy from 0.05% of the parameters.

On Countdown and MATH, FuRA-S is on par with full-parameter ES while training at most 0.05% of the parameters it perturbs. It outperforms ES over LoRA and VeRA adapters on every model, and without computing a single gradient it surpasses PPO, GRPO and Dr. GRPO on four of five models.

Accuracy by model and benchmark

Training the released INT4 model

Countdown accuracy of the GPTQ-INT4 checkpoints over training, with the 4-bit weights left unchanged. Quantized ES (QES) perturbs the quantized weights themselves; the dotted line is its best accuracy over the whole run.

FuRA-S on INT4 (ours)QES

FuRA-S values from the paper’s table; QES read from its figure, matching the reported best and final accuracies. Serving a specialist on the INT4 kernels adds about 9% memory and 11–15% decoding time per token.

On device

An 8B specialist, trained on an iPhone overnight.

On an iPhone 16 Pro, MobileSpec trains a specialist for one LaMP movie-tagging user on top of the released INT4 Llama-3.1-8B-Instruct, using the likelihood of the user's own records as fitness. The 5.5 GB model is larger than the 5.3 GB an app may use, so MobileSpec streams the decoder layers from flash.

Twenty iterations, 10.4 hours, one phone

Training and validation loss over wall-clock time on the phone, and where the memory goes. Hover the curves.

validationtrainingbase model

Validation loss falls from 0.85 to 0.19, and the specialist answers 8 of 15 held-out queries correctly against 5 for the generalist. Forward computation takes 97% of the time; streaming the layers takes 14 seconds per iteration. Curves read from the paper’s figure.

Personalization with the user's own records

LaMP with Llama-3.1-8B-Instruct in the time-based setting: each user gets their own specialist, trained on earlier records and tested on later queries. Per-user LoRA is trained by backpropagation; FuRA-S uses forward passes only and stores 4.98 MB per user. Best per row in bold.

FAQ

Questions people ask

What does MobileSpec do?

It keeps one shared, quantized LLM on the phone (the generalist) and trains a small specialist for each task the user repeats, on the phone itself while it is idle. Training uses forward passes only and learns from signals that arise during use: automatic checks of a task's outcome, or the likelihood of the user's own records.

Why not train with backpropagation or LoRA on the phone?

Backpropagation needs several times the memory of inference and is poorly supported by the quantized dataflow of on-device engines. Forward-only methods avoid that cost, but full-parameter evolution strategies keep a full-precision copy of the weights and make every specialist as large as the model, and LoRA's generic low-rank perturbations soon stop finding improving directions under ES.

What are block-spectral coordinates?

Each weight matrix is split into column blocks, and each block is factored by an untruncated SVD, so W_k = L_k diag(s_k) R_k^T reconstructs the weights exactly. FuRA-S trains only the block singular values s_k (about 0.03% of the parameters of Qwen2.5-1.5B) and freezes L_k and R_k.

Why does searching such a tiny space work?

RLVR mostly rotates the singular vectors of pretrained weights while preserving their singular values (spectral inheritance). Rescaling the singular values of each block independently shifts the blocks' relative contributions to the assembled matrix, which rotates its singular vectors while barely changing its singular values. ES therefore keeps finding improving perturbations for longer, and the rotated frames alone retain over 95% of the accuracy gain.

How does it run on an INT4 model?

Because nothing is truncated, the update can be rewritten exactly as W' = W(I + R diag(delta) R^T), a transformation of the layer input. The released INT4 weights, their scales and zero points, and the existing INT4 kernels stay untouched; each specialist stores only its full-precision delta, and delta = 0 recovers the quantized model exactly.

How well does it work?

On five MobileWorld phone-use tasks, specialists of MAI-UI-2B reach 64–100% success from a 6–23% base. On RLVR benchmarks FuRA-S is competitive with full-parameter ES while training at most 0.05% of the parameters, outperforms ES over LoRA adapters, and beats quantized ES on INT4 models by up to 16.25 points. On an iPhone 16 Pro it trains a personalized Llama-3.1-8B specialist in 10.4 hours with 3.5 GB peak memory.

Citation

BibTeX

Abstract

Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1–5 MB that update 0.015–0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6–23% up to 89–100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.

@misc{tan2026mobilespec,
  title         = {Training Specialist {LLMs} on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates},
  author        = {Tan, Liyan and Zhao, Yequan and Liu, Ziyue and Su, Yupeng and Zhang, Ruijie and Zhang, Zheng},
  year          = {2026},
  eprint        = {2611.04797},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {Liyan Tan and Yequan Zhao contributed equally}
}