Thibault Rousset¹, Taisei Kakibuchi², Yusuke Sasaki² and Yoshihide Nomura², ¹McGill University, Canada, ²Fujitsu Research Ltd, Canada

Advancements in Natural Language Processing have enabled specialized language models, but integrating domain-specific knowledge into general-purpose models in multilingual settings remains challenging, particularly for technical vocabulary. This paper explores cross-lingual knowledge transfer in model merging, examining how combining a general-purpose language model with a domain-specific model affects technical jargon comprehension. The objective is to evaluate the effectiveness of merging techniques in enhancing domain-specific proficiency while preserving general language understanding. Our study analyzes different merging strategies and their impact on specialized terminology retention. A quantitative evaluation compares the merged model’s performance against its constituent models, offering insights into the strengths and limitations of various approaches. The results demonstrate the potential of model merging for domain adaptation while highlighting challenges in cross-lingual knowledge transfer. These findings provide valuable guidance for optimizing model merging techniques in specialized NLP applications.

Large Language Models, Knowledge Transfer, Model Merging, Domain Adaptation, Natural Language Processing