The State of Uzbek Data for AI

A Comprehensive Overview of Open-Source Uzbek Datasets and Their Roles in LLM Development

1. Introduction

As artificial intelligence continues to reshape industries worldwide, language remains the foundation of intelligent systems. For Uzbekistan, the future of AI depends on the ability of models to understand, reason, and communicate in Uzbek—a rich language with a rapidly digitalizing presence but still limited representation in machine learning datasets.

While English, Chinese, and other models benefit from vast open corpora, Uzbek data remains fragmented and under-structured. This article provides a detailed exploration of available Uzbek datasets, classifies them by type, and explains how each class contributes to building modern large language models (LLMs) and NLP models. It also presents an exploratory data analysis (EDA) of open-source Uzbek data to identify what exists today and what is still missing for true Uzbek AI capability.

2. Why Uzbek Data Matters

Training a large language model is not only about collecting words - it is about teaching language comprehension, task understanding, and reasoning. Each category of dataset serves a distinct function in that process:

• Raw text corpora teaches grammar, structure, and contextual flow.
• Instruction datasets teach models to follow human directions and respond appropriately.
• Chain-of-Thought (CoT) data develop step-by-step reasoning and explainable logic.
• Annotated datasets train and evaluate specific skills such as classification or information extraction.
• Evaluation datasets are used to measure performance across language understanding, reasoning, and instruction-following tasks. They provide standardized benchmarks for comparing LLMs.

Understanding how much data exists in each category helps organizations and researchers design the next generation of Uzbek-capable AI systems.

3. Classes of Uzbek Data

Based on an examination of open repositories such as Hugging Face, Mendeley Data, and academic corpora, Uzbek data can be grouped into four principal classes:

4. EDA of Open-Source Uzbek Datasets

In linguistic coverage, Uzbek data is sufficient for language modeling: large corpora-like uz-crawl and uz-books ensure grammatical and lexical diversity. However, in reasoning and structured task learning, the resources are minimal. Models trained purely on existing data can speak Uzbek but cannot reason in Uzbek—they lack exposure to instructional patterns, logic sequences, and structured decision-making examples.

5. How Each Class Contributes to Model Training

Without instruction and CoT data, a model trained solely on text will appear fluent yet behave superficially—unable to follow directives, justify decisions, or apply rules.

6. Key Findings from the EDA

1. The Uzbek data ecosystem is text-rich but behavior-poor.
   Most datasets contain free text rather than supervised interactions or reasoning traces.
2. Instructional data is emerging but not domain-aligned.
   Current examples are general rather than specialized (e.g., law, education, healthcare).
3. Reasoning (CoT) data remains the largest gap.
   No open Uzbek dataset provides reasoning or explanation examples—limiting higher-order understanding.
4. Annotated datasets serve auxiliary purposes.
   They are useful for evaluation but cannot teach complex cognitive behavior.

7. Implications for Future Uzbek AI Development

The current landscape demonstrates that Uzbekistan has made progress in linguistic data availability, but still lacks reasoning-oriented, instruction-rich corpora.

To advance national AI research and productization:

• Develop domain-specific instruction datasets (e.g., legal Q&A, civic information, educational tutoring).
• Create Chain-of-Thought datasets in Uzbek that show intermediate reasoning steps and explanations.
• Establish standardized benchmarks to measure understanding, reasoning, and factual correctness in Uzbek.
• Encourage open collaboration between universities, AI labs, and government agencies to share curated, structured data.

8. Translation Data for Uzbek ↔ English

8.1 What Translation Data Is

Translation datasets are called parallel corpora — collections of sentence pairs (or paragraphs) in two or more languages that express the same meaning.

Each record has the structure:

{
  "source": "Men Toshkentda yashayman.",
  "target": "I live in Tashkent."
}

They are used to train and evaluate neural machine translation (NMT) models such as MarianMT, M2M100, or NLLB (No Language Left Behind).

In contrast to raw text corpora, which teaches language patterns, parallel data explicitly teach cross-lingual alignment — how a sentence in Uzbek corresponds semantically and syntactically to one in English (or another language).

8.2 Types of Translation Data

There are three main categories of translation datasets relevant to Uzbek:

8.3 Overview of Available Uzbek Translation Resources

8.4 How These Datasets Are Used

a) For Traditional Machine Translation (NMT)

Parallel Uzbek–English corpora are used to train or fine-tune models like:

• MarianMT, mBART, NLLB, or M2M100
• Objective: learn direct translation mapping via supervised sequence-to-sequence modeling.

b) For Multilingual LLMs and Instruction Fine-Tuning

Translation pairs can also be reframed as instruction tasks to fine-tune multilingual LLMs.

Example format:

{
  "instruction": "Translate the following Uzbek sentence into English.",
  "input": "Bu qonun 2023-yil 15-iyulda qabul qilingan.",
  "output": "This law was adopted on July 15, 2023."
}

This method doesn’t just teach the model bilingual equivalence — it also teaches how to perform translation on request, integrating translation into the model’s conversational behavior.

c) For Alignment and Multilingual Embeddings

Some Uzbek–English corpora are used to train multilingual embedding models (like LaBSE, LASER3).
These models learn a shared vector space where semantically similar sentences in both languages have nearby representations.
This is essential for multilingual search, cross-lingual RAG systems, or translation memory tools.

8.5 Quality and Coverage Challenges

Despite progress, Uzbek translation datasets face several limitations:

1. Data Size — Most Uzbek↔English corpora are small (tens of thousands of pairs), compared to tens of millions for major languages.
2. Domain Imbalance — Many pairs come from Wikipedia or news sources; there’s limited legal, technical, or colloquial data.
3. Alignment Noise — Automatically aligned datasets (like CCAligned) contain significant mismatches due to inconsistent sentence segmentation.
4. Script Variation — Uzbek appears in both Latin and Cyrillic scripts, complicating alignment and requiring preprocessing.
5. Cultural and Legal Context Loss — Direct translations often omit context critical for Uzbek-specific meaning (e.g., legal or bureaucratic expressions).

8.6 Summary of Translation Data Usefulness

8.7 Outlook

To advance translation quality and integrate it with high-level reasoning (e.g., bilingual legal assistants), future projects should:

• Expand professionally curated bilingual corpora with domain diversity (law, healthcare, government).
• Add reasoning-style translation datasets, showing how to translate nuanced or context-heavy Uzbek expressions step-by-step.
• Develop script normalization tools to handle Cyrillic↔Latin consistency.
• Create translation benchmarks specific to Central Asian languages for fair evaluation (BLEU, COMET, ChrF++).

With these improvements, Uzbek can move from being a “low-resource language” to a strategically multilingual language in global AI.

9. Conclusion

Uzbek-language AI development has entered a transformative stage.
In its early years, progress centered on text collection — web crawls, literary archives, and digitized books that gave language models their foundational linguistic understanding.
Today, the next frontier lies not in more text, but in structured and functional data: resources that teach models not only how Uzbek sounds, but how Uzbek thought and reasoning operate.

Through this analysis, it is clear that:

• Raw text corpora forms a solid linguistic base for pretraining.
• Instruction datasets (like UAzimov/uzbek-instruct-llm and Behbudiy/alpaca-cleaned-uz) provide the first step toward task-following behavior.
• Annotated datasets add evaluation capabilities for classification and named-entity recognition tasks.
• Chain-of-Thought data—the foundation of reasoning and interpretability—remains a critical missing piece.
• Translation datasets (e.g., Behbudiy/translation-instruction, OPUS Tatoeba, WikiMatrix) are becoming an essential bridge between Uzbek and global languages, enabling multilingual LLMs and cross-lingual retrieval systems.

The strategic direction for the Uzbek AI community is now clear:

• Develop domain-specific instruction datasets (especially in law, healthcare, and education).
• Create reasoning-rich Chain-of-Thought corpora that capture Uzbek analytical processes.
• Expand translation and alignment data to ensure interoperability between Uzbek and other major languages.
• Build national evaluation benchmarks that measure reasoning quality, factual accuracy, and multilingual performance.
• Establish Uzbek GLUE benchmark dataset (UzGLUE) to train LLMs in different tasks.

By shifting focus from raw data collection to structured, high-quality, and explainable datasets, Uzbekistan can evolve from a low-resource language environment into a regional AI innovator—one that promotes ethical, transparent, and locally grounded artificial intelligence aligned with the country’s cultural and linguistic identity.

References

• UAzimov / uzbek-instruct-llm — Hugging Face
• Behbudiy / alpaca-cleaned-uz — Hugging Face
• Behbudiy / translation-instruction — Hugging Face
• Behbudiy / uzbek-sentiment-analysis — Hugging Face
• Tahrirchi / uz-crawl — Hugging Face
• Tahrirchi / uz-books — Hugging Face
• Parallel Uzbek–Kazakh Corpus — PMC
• SimRelUz — arXiv
• NER for Uzbek — Mendeley Data
• OPUS Tatoeba — University of Helsinki
• WikiMatrix — Facebook AI