NVIDIA details 2.8-trillion-token Indic corpus and open pipeline

NVIDIA and AI4Bharat described a forthcoming Indic-language pretraining corpus containing about 2.8 trillion tokens across 22 Indian languages. NVIDIA’s Nemotron overview lists about 2.83 trillion synthetic output tokens across 34 language-script configurations, while its training-recipe documentation says the corpus was announced for release. The public generation recipe is available now, but those documents do not present the corpus itself as already downloadable.

The corpus design covers all 22 languages in native script. For 12 priority languages, the workflow also produces romanized, Latin-script records, creating the 34 language-script configurations. Romanization is a second stage: the system transliterates an accepted native-script translation and evaluates that result again with an added script-compliance check.

The public NeMo Data Designer recipe says the workflow starts from the English Nemotron-CC v2 corpus. Its plugin translates documents into any of the 22 languages, then scores outputs against the source for faithfulness, fluency, terminology, style and format preservation. Scores use a 0–100 rubric. Where rewriting is enabled, an output below the configured threshold can be rewritten and rescored, with superseded attempts retained in history fields.

For Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu, Gemma-4-31B-it handles translation and evaluation. Those are the same 12 languages that receive romanized variants.

The pipeline routes eight other languages—Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Sanskrit and Sindhi—through the Bodhan Indic Translate specialist and uses Gemma-4-31B-it as evaluator. Automatic rewriting is disabled on that route. NVIDIA’s documentation says the specialist’s required prompt format does not support the rewrite loop and that rewrites for these languages tended to produce repetition.

Santali and Manipuri take separate paths. Long Santali documents can be split into overlapping windows of up to about 2,048 tokens and stitched back together before whole-document evaluation. Manipuri is translated into Bengali script, deterministically remapped to Meitei Mayek and evaluated again after the remap.

These routes expose stated limitations in the generation process. NVIDIA says no single model performs best across all 22 languages, direct generation in Meitei Mayek was unreliable, and long Santali translations degraded or entered repetition loops. The documentation also notes that its native-numeral option records what the model was asked to do but does not verify that native digits were actually produced. NeMo Data Designer’s v0.9.5 release notes record the addition of the developer note for the translation plugin.