Can AI Actually Speak Nepali? We Stress-Tested the Big Models on Devanagari

The test nobody publishes

Global benchmarks celebrate multilingual scores, but Nepali hides inside a footnote labelled other languages. Fluency on encyclopaedia prose is easy; the hard cases live in daily life — converting Bhadra 15, 2082 into a Gregorian date correctly, transliterating a Preeti-font court order, or explaining why bhaat ramro means more than rice being good.

So we built an informal battery: 120 prompts across five categories, scored blind by two native speakers on a simple 0-to-2 scale. This is not an academic benchmark with error bars — treat it as a field report from people who use these tools every working day.

The harness we used

import json, requests PROMPTS = json.load(open("nepali_prompts.json")) def ask(model: str, prompt: str) -> dict: r = requests.post( f"https://api.{model}.example/v1/chat", json={"messages": [{"role": "user", "content": prompt}]}, timeout=30, ) return {"model": model, "reply": r.json()["content"]} # Blind scoring happens out-of-band: replies are shuffled and graded # 0-2 by native reviewers before any model label is revealed. for p in PROMPTS: for m in ["frontier", "open-mid", "nepali-ft"]: print(ask(m, p["text"])) Every frontier model we tested wrote fluent Nepali paragraphs — then confidently converted 2082 BS to 2024 instead of 2025/26 AD, or invented a thirty-day Bhadra. Calendar errors are invisible to non-Nepali evaluation sets and embarrassing in production.

What the scores showed

Two findings matter more than the averages. Models fail loudly on scripts they barely saw — legacy encoding goes in, garbage comes out — but fail silently on calendars, producing confident, well-formed nonsense. And code-switching, the way Kathmandu actually talks, beat pure-Nepali tasks everywhere, betraying training data heavy on English scaffolding.

It writes like a newsreader who learned the language from headlines — perfect grammar, no grandmother.

What would actually fix it

Quantity is not the bottleneck; curation is. A few thousand cleaned sentence pairs for BS-calendar arithmetic, a public Preeti-to-Unicode corpus, and idiom dictionaries crowdsourced from social-media comments would move scores further than another billion parameters. Nepal has the linguists and the data owners — what it lacks is coordination.

• Fluent Nepali output masks brittle factual layers — always verify dates and numbers independently.
• Calendar conversion (BS to AD) is the highest-value fine-tuning target for Nepali AI.
• Open-weight models trail frontier ones narrowly on translation but collapse on legacy fonts.
• Benchmarks that ignore code-switching misrepresent how Nepalis actually use language.

Anish — building in Nepali NLP@kaagaz_dev

Asked four LLMs what date Shrawan 15, 2082 falls on. Got four different answers. Three were wrong. The fourth said it was not certain — instantly my favourite.

Field notes · this month

♥ 842⟳ 197

AI tools promise Nepali-language access but stumble on cultural context नेपाली भाषामा एआई: अवसर र चुनौती Local startups race to build Nepali-first chatbots as global models lag