SEA-LION v4 is an open multimodal model trained for Southeast Asian languages and cultural contexts

The team found the Gemma 3 tokenizer required no modifications, providing robust performance on SEA language datasets in assessments.

Data preparation and curation

The team’s approach to data selection was split between curating a broad knowledge base for continued pre-training and crafting precise, skill-oriented data for post-training.

Selecting data for continued pre-training (CPT)

Enriching the model’s core knowledge, particularly in lower-resource languages, required a vast, clean dataset.

• Data cleaning: Deduplication and sophisticated filtering using a combination of heuristics and an “LLM-as-a-judge” approach to remove low-quality content and toxic language, validated by initial small-batch reviews with native speakers
• Data tagging: Quality tagging system to classify data tiers (e.g., ‘high’, ‘normal’, ‘low’) and enable quality-aware data mixing, where higher-quality data (like research papers) is strategically weighted over lower-quality sources (like casual forum posts). This boosted performance in Indonesian and Tagalog without causing regressions elsewhere

Experiments confirmed that Gemma 3 had likely already seen high-quality public data for high-resource languages. Consequently, we found that prioritizing novel, “unseen” data for languages like Indonesian yielded better results than retraining on “seen” data.

Crafting data for post-training

Post-training is where the model learns to be helpful, follow instructions, and respond safely. High-quality instruction-following data in many SEA languages is scarce, making synthetic data generation a cornerstone of model alignment.

• From CPT data: A multi-step AI agent pipeline was built to transform CPT documents into high-quality multi-turn conversational data. The agent checks if a document could lead to a conversation, creates conversation outlines, drafts the dialogue, and reviews quality.
• Preference data from supervised fine-tuned (SFT) conversations: SFT conversations were used to create “better/worse” preference pairs for preference tuning, identifying areas where the assistant’s response was strong and then rewriting it to be intentionally worse

Maximizing efficiency for experimentation

GPU compute is a scarce and expensive resource and with strict timelines, the quality of training comes down to how effectively one can navigate the complex experimental landscape. The team focused on maximizing insights from every run and prioritizing rapid, small-scale experiments with smaller models like Gemma 3 1B and 4B before scaling successful approaches to the larger, more resource-intensive 27B model.

Vertex AI provided a customized and optimized version of Nvidia’s NeMo Framework, which delivered a significant 13-23% throughput improvement over Composer. Working with Vertex AI engineers, the team further optimized the NeMo container for Gemma 3 and improved training step latency by 1.96x.

Training

The primary challenge during continued pre-training was to teach the model new linguistic and cultural nuances without losing pre-existing capabilities. This required a meticulous data mixing strategy, carefully balancing new data with the model’s original training corpus. Methods like model merging also offered a powerful way to combine a specialized model’s knowledge with the base model’s general capabilities.

Once the model has sufficient knowledge, post-training aligns it to be a more helpful assistant that provides responses that are more grounded in local contexts. The post-training phase began with supervised fine-tuning on instruction-response pairs, then progressed to alignment algorithms like Direct Preference Optimization (DPO) and Reinforcement Learning (RL) to align the model outputs closer to human preferences.

Executing these steps effectively is where purpose-built frameworks like NVIDIA’s NeMo-RL are essential, providing optimized and scalable implementations for the state-of-the-art alignment techniques.

Building a comprehensive evaluation suite

A cohesive evaluation strategy is paramount for guiding model development and must be adapted to the specific nuances of the target use case. SEA-LION v4 was evaluated on four pillars.

1. Training metrics

During training, the team developed an understanding of where the training was heading by looking at the loss curve, identifying inflection points, and checking if metrics aligned with expectation.

Additional metrics like perplexity-based and log probability-based evaluations provided deeper insights into the model’s fundamental language and sequence understanding, enabling informed decisions earlier in the training process.

2. Benchmarks

For English, the team evaluated the model on BBH, GPQA, IFEval, Math-Hard, MMLU-Pro, and MUSR tasks to ensure the model’s core reasoning, understanding, and instruction-following capabilities remained world-class.

For SEA languages, they used the SEA-HELM benchmark suite which tests performance for Burmese, Filipino, Indonesian, Malay, Tamil, Thai and Vietnamese across a diverse range of tasks, such as natural language processing, chat, and instruction-following.

3. Synthetic evaluation data

To address the signal gap in pre-training data for low-resource languages, novel evaluation sets for Burmese and Khmer were generated using Vertex AI’s synthetic data capabilities. This approach ensured these languages had reliable metrics for measuring progress.

4. Human-in-the-Loop validation

The team worked with native speakers to understand the model’s performance with the depth and detail that only a native speaker could provide. Standardizing these metrics allowed them to evaluate the model with fine-grained understanding in low-resource language settings.

Model lifecycle & operations

A state-of-the-art model is not a static artifact but a living project. Our development philosophy treats every training run as a formal experiment, meticulously tracking metrics to compare different training methods and validate hypotheses. Each successful experiment that yields a superior model culminates in a new, clearly documented version.

This rigorous model versioning is crucial not only for reproducibility and traceability but also for managing a continuous cycle of retraining that ensures the model’s capabilities consistently evolve and improve over time.