Proposes a novel approach to the important problem of data mixture optimization for continual pre-training. Instead of exploring different data mixes, it optimizes the merging of model updates from individual datasets, demonstrating improved performance at substantially lower search cost.
Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: it must be fixed before training begins, and a suboptimal choice can waste weeks of compute.
In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15–35× lower search cost.
Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.
Instead of fixing the data mixture ratio before training, we train a separate continual pre-training model on each dataset independently and extract the corresponding distribution vector (the parameter shift from the base pre-trained model) after training.
OptiMer is 15–35× faster than data mixing for searching optimal ratios.
OptiMer achieves the highest average score across all dataset combinations, outperforming the data mixing baseline in each group by 2.1–6.7 points.
OptiMer can produce an objective-optimized model on demand without retraining.
Continual pre-training and instruction-tuned vectors are nearly orthogonal (cosine ≈ 0.03), and different continual pre-training vectors also exhibited low similarity (0.29–0.31), thus they can merge with low conflict.
Principal component analysis of the same vectors (sparsified through layer-wise truncated singular value decomposition); the accompanying bar charts show the optimal merge weights. Both confirm that continual pre-training vectors lie far from the instruction-tuned vector.
Weight combinations with high scores are concentrated in a narrow region with a large instruction-tuned weight and small continual pre-training weights; this sharp optimum makes grid search impractical, whereas the Tree-structured Parzen Estimator quickly approaches the promising region and focuses exploration within it, and OptiMer converges within 100 trials.
The continual pre-training trajectory is approximately linear, indicating that adjusting the merge weight is analogous to controlling the effective training duration.
We apply OptiMer to the Gemma-SEA-LION-v4-27B model, a Gemma 3 based model pre-trained on Southeast Asian languages: it improves the Japanese leaderboard score from 66.34 to 74.40 and the overall average from 54.37 to 70.19 over the base SEA-LION-v4-27B-IT, while maintaining the ASEAN language task performance.
In this paper, we present OptiMer, a framework that decouples data mixture ratio selection from model training in continual pre-training. OptiMer trains separate continual pre-training models per dataset and composes their distribution vectors with weights optimized via Bayesian optimization. Experimental results show OptiMer consistently outperforms data mixture baselines while requiring 15–35× lower search time. We further demonstrate OptiMer's flexibility to produce multiple objective-tailored models from a single vector pool without retraining.
Our work suggests that data mixture ratio selection in continual pre-training can be reformulated as a post-hoc optimization problem instead of being fixed before training. Future work includes extending OptiMer to the mid-training, where deciding the mixture ratio of a large number of domain-specific datasets remains a central challenge.
Haiyue Song, Huy Hien Vu, and Masao Utiyama. 2026. OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 25034–25054, Budapest, Hungary. Association for Computational Linguistics.
@inproceedings{song-etal-2026-optimer,
title = "{O}pti{M}er: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training",
author = "Song, Haiyue and
Vu, Huy Hien and
Utiyama, Masao",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1090/",
pages = "25034--25054",
ISBN = "979-8-89176-459-0"
}