OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

Abstract

Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: it must be fixed before training begins, and a suboptimal choice can waste weeks of compute.

In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15–35× lower search cost.

Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.

Method

Instead of fixing the data mixture ratio before training, we train a separate continual pre-training model on each dataset independently and extract the corresponding distribution vector (the parameter shift from the base pre-trained model) after training.

Results

OptiMer is Efficient

OptiMer is 15–35× faster than data mixing for searching optimal ratios, and this advantage becomes larger with more datasets.

OptiMer is Effective

OptiMer achieves the highest average score across all dataset combinations, outperforming the data mixing baseline in each group by 2.1–6.7 points.

OptiMer is Flexible

OptiMer can produce an objective-optimized model on demand without retraining.

Analysis

Why Mergeable?

Continual pre-training and instruction-tuned vectors are nearly orthogonal (cosine ≈ 0.03), and different continual pre-training vectors also exhibited low similarity (0.29–0.31), indicating that each distribution modifies an independent subspace, which supports the feasibility of linear composition.

Principal Component Analysis of
Distribution Vectors

Principal component analysis of the same vectors (sparsified through layer-wise truncated singular value decomposition); the accompanying bar charts show the optimal merge weights. Both confirm that continual pre-training vectors lie far from the instruction-tuned vector.

OptiMer Search Dynamics

Weight combinations with high scores are concentrated in a narrow region with a large instruction-tuned weight and small continual pre-training weights; this sharp optimum makes grid search impractical, whereas the Tree-structured Parzen Estimator quickly approaches the promising region and focuses exploration within it, and OptiMer converges within 100 trials.

Continual Pre-Training Dynamics

The continual pre-training trajectory is approximately linear, indicating that adjusting the merge weight is analogous to controlling the effective training duration, which also explains why OptiMer assigns small continual pre-training weights.

Distribution vector trajectories during continual pre-training

Application

Generalization to the SEA-LION Model

We apply OptiMer to the Gemma-SEA-LION-v4-27B model, a Gemma 3 based model pre-trained on Southeast Asian languages: it improves the Japanese leaderboard score from 66.34 to 74.40 and the overall average from 54.37 to 70.19 over the base SEA-LION-v4-27B-IT, while maintaining the ASEAN language task performance.

Conclusion

In this paper, we present OptiMer, a framework that decouples data mixture ratio selection from model training in continual pre-training. OptiMer trains separate continual pre-training models per dataset and composes their distribution vectors with weights optimized via Bayesian optimization. Experimental results show OptiMer consistently outperforms data mixture baselines while requiring 15–35× lower search time. We further demonstrate OptiMer's flexibility to produce multiple objective-tailored models from a single vector pool without retraining.

Our work suggests that data mixture ratio selection in continual pre-training can be reformulated as a post-hoc optimization problem instead of being fixed before training. Future work includes extending OptiMer to the mid-training, where deciding the mixture ratio of a large number of domain-specific datasets remains a central challenge.

How to Cite

We will update the citation after the EMNLP 2026 proceedings are public.

bibliography

Haiyue Song and Masao Utiyama. “OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training.” arXiv preprint arXiv:2603.28858 (2026).

bibtex

@misc{2603.28858,
  Author = {Haiyue Song and Masao Utiyama},
  Title = {OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training},
  Year = {2026},
  Eprint = {arXiv:2603.28858},
}