Covering native-script data for 22 languages, romanized data for 12 languages, and a reusable GPU-accelerated pipeline for translation, quality assessment, and remediation.
Authors: AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.
We are thrilled to introduce Nemotron-pretraining-Indic-v1, a dataset spanning 22 Indic languages and ~2.8 trillion tokens, to be released alongside the open-source GPU-accelerated data generation pipeline used to build it. All 22 languages are recognized under the Constitution of India: the initial batch of languages…