AI4Bharat Launches “Indic LLM-Arena” to Benchmark Language Models for India’s Diverse Linguistic Needs

AI4Bharat (IIT Madras) has unveiled Indic LLM-Arena, a crowd-sourced, human-in-the-loop leaderboard created to evaluate large-language models (LLMs) on Indian-specific criteria of language, context, and safety.

Recently, OpenAI unveiled a new benchmark called IndQA designed to evaluate how well AI models understand and reason about Indian languages and culture.

The announcement by AI4Bharat addresses a crucial gap: most global AI benchmarks are heavily English-centric and do not account for India’s multilingual, code-switched and culturally varied user base.

“A model’s ability to discuss a topic in perfect English is irrelevant if it fails to understand a farmer in rural Maharashtra, provides a culturally inappropriate response to a user in Sikkim, or cannot parse a Tang-lish query from a student in Tamil Nadu,” states the initiative.

The platform emphasises three core gaps: the language gap (code-mixing across Indian languages), the contextual/cultural gap (regional correctness of responses), and the safety/fairness gap (biases unique to the Indian social fabric).

Indic LLM-Arena allows users to enter prompts in any Indic or mixed language, compare anonymised responses from two different models, vote on which performs better, and contribute to the community-driven ranking.

The system uses the Bradley-Terry statistical model to establish rankings based on thousands of user-signed “battles.”

The initiative offers value across stakeholders: developers get benchmark data tailored for India-use cases, enterprises gain decision-making tools for model selection, and the public benefits from more inclusive, context-aware AI.

Moving forward, AI4Bharat plans to expand into vision, audio, agentic tasks, open leaderboards and open-source datasets.

“The Indic LLM-Arena is an open invitation to the entire community to help us define what ‘good’ AI looks like for India,” the blog concludes.