AdoptingAI Issue 02: The Token Tax on Indian Languages

AI is priced by the token, and Indian languages need more of them. We measured nine of today's models, including OpenAI's GPT-5.6, Google's Gemma 4 and Meta's Muse Glimmer. On every one, the same sentence costs more in every Indian language than in English: from 1.53x to 7.11x, depending on the model.

Why Indian languages cost more

On 16 October in Bengaluru, OpenAI made India's languages its pitch: new Hindi voices, two-way Hindi conversation in ChatGPT and models trained on Hinglish. The race to sell AI to India in its own languages is on. This issue is about the meter running underneath it.

AI models don't read words. They read tokens, fragments of text cut from a fixed vocabulary, and every commercial model bills, remembers and answers in tokens. Vocabularies learned mostly from English cut English into big pieces and Indian scripts into small ones. Here is what that looks like on OpenAI's current tokenizer:

Punjabi needs nearly four times the tokens of English for the same sentence. The market paying that premium is large: India has 958 million active internet users (IAMAI, January 2026), 57% of users prefer content in Indian languages (Kantar and IAMAI, January 2025), and ChatGPT alone has about 100 million weekly users in India (February 2026).

What we measured

On 20 October 2026, AdoptingAI ran Articles 1–30 of the Universal Declaration of Human Rights, in English and nine Indian languages, through the tokenizers of nine models: OpenAI's GPT-5.6, Google's Gemma 4, Meta's Muse Glimmer, Alibaba's Qwen3.8, DeepSeek's V4.1 Flash, NVIDIA's Nemotron 3.5 Super, Xiaomi's MiMo-V2.6, Z.ai's GLM-5.3 and Sarvam AI's 105B. Each score is tokens needed divided by tokens needed for English on the same tokenizer, and each model's average is a simple mean across the nine languages. Figures are for the specific checkpoints named in the methodology. Every number in this issue comes from our own run unless a source is cited. Models we couldn't test, including GPT-6, Gemini and Claude, are listed in the methodology with what public sources say about them.

Three findings

1. Choosing a model is choosing a tax rate. No model in our test charges Indian languages the same as English. Averaged across nine languages, the premium runs from 1.53x on Sarvam 105B to 7.11x on GLM-5.3. On GLM-5.3, a Tamil sentence needs 9.72 times the tokens of its English original. These are token multiples measured within each model, not dollar prices: per-token prices differ between models, so a higher premium does not always mean a bigger bill.

2. This year's launches mostly left the tax alone. Most of the 2026 releases we tested kept their predecessor's vocabulary. In our test, Meta's Muse Glimmer gives exactly the same counts as Llama 4. DeepSeek's V4.1 Flash, updated on 17 October, matches V4 to the token. Xiaomi's September MiMo release gives exactly the same counts as Qwen 3's old vocabulary. The exception is Alibaba: Qwen3.8's vocabulary (we tested the 27B checkpoint) brings the average premium down to 2.92x, from 6.42x on Qwen 3, which shows the tax is a design choice.

3. Even the lowest premium is still a premium. The lowest average in our test, 1.53x against 1.78x for Gemma 4 31B, came from Sarvam AI's 105B, a model that Sarvam says covers India's 22 scheduled languages. Its counts match Gemma 4's exactly for five of the nine languages and are lower for Gujarati, Kannada, Malayalam and Punjabi. It still needs more tokens than English in every language we tested, up to 1.86x for Tamil.

What the tax costs in practice

Per-token prices are the same in every language, so a token premium is a cost premium. Here is the gap on four everyday jobs, using the premiums we measured on OpenAI's GPT-5.6 tokenizer and an illustrative price of $10 per million input and $50 per million output tokens:

The costliest effect is the least visible. A model's memory, its context window, is fixed in tokens, so a Tamil document fills it faster. A 128,000-token window holds about 229 pages of English on GPT-5.6, but about 91 pages of Tamil. A 150-page court judgment fits in one go in English; in Tamil it has to be split, and the model can lose the thread between parts. On Z.ai's GLM-5.3, released in August, the same window holds just 24 pages of Tamil.

Why this is new for software

For thirty years, building software for India meant translating the interface once. After that, a user in Chennai cost the same to serve as a user in Chicago. AI breaks that. Every time the software thinks, it is billed by the token, and the tokenizer decides what your language costs. Language has moved from the localisation budget to the cost of goods sold.

In their words

Researchers flagged this years ago. Today it is a growth problem for everyone selling AI to India.

What this means

For CXOs and CIOs. Add one line to every AI evaluation: how many tokens does our real workload take, in our customers' languages? A model that wins an English benchmark can lose your budget in Tamil.

For founders. Unit economics built on English token counts can break in Indian languages. Pick your model stack by language mix, and route traffic: the most token-efficient model for each language.

For investors. Ask Indian AI companies for cost per interaction by language. The gap between a 1.5x and a 7x tokenizer can decide whether a business has margins at all.

For policymakers. Procure AI for citizen services on cost and context capacity in the languages citizens use. A public tokenizer benchmark for all 22 scheduled languages would be cheap to build: the entire test for this issue ran in a web browser.

The closing thought

We usually describe India's AI gap as a gap in chips, models or talent. This one is smaller and more fixable: missing pieces in a vocabulary file. Alibaba's newest vocabulary shows it can be cut. India's next 500 million AI users are unlikely to type in English. Whoever prices AI fairly for them will reach them first.

Methodology and sources

Method. We used the body text of UDHR Articles 1–30 from the UDHR in Unicode collection distributed with NLTK, excluding headings, and counted tokens without special tokens. Open-model tokenizers were loaded on 20 October 2026 from each model's official Hugging Face repository (Qwen/Qwen3.8-27B, meta-models/Muse-Glimmer-30B, nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, deepseek-ai/DeepSeek-V4.1-Flash-1017, XiaomiMiMo/MiMo-V2.6-Pro-RL, zai-org/GLM-5.3, google/gemma-4-31b-it, sarvamai/sarvam-105b) using Hugging Face's transformers.js, and cross-checked against our earlier Python counts. OpenAI's counts use the o200k tokenizer via gpt-tokenizer v4.1.0, which maps GPT-5 through GPT-5.6 to o200k.

Limitations. OpenAI's model-to-tokenizer mapping comes from the open-source gpt-tokenizer library, not from OpenAI. The UDHR is formal text, and conversational text may tokenize differently. Token counts measure cost and capacity, not answer quality.

Models we could not test, and what public sources say. We could not run these ourselves, so none of our measurements are for them. Where an independent source has published tokenizer information, we note it. The main source is ellamind's Tokenization Tax Report (7 September 2026), which counted tokens through providers' own APIs across 12 languages, none of them Indian.

OpenAI GPT-6. OpenAI hasn't published GPT-6's tokenizer. ellamind reports that GPT-6 Astra's API counts matched OpenAI's o200k tokenizer, the one we measured for GPT-5.6. If that holds, our GPT-5.6 figures apply to GPT-6 Astra.
Google Gemini 3.x. Google doesn't publish Gemini's tokenizer. ellamind reports that Gemini 3.x and Gemma 4 share the same 262,144-token vocabulary and that Gemini's API counts matched local Gemma 4 counts. If that holds, our Gemma 4 figures apply to Gemini 3.x. We found no public information on Gemini 4 Argon (October 2026).
Anthropic Claude (Opus, Sonnet and Haiku 5.5). Anthropic doesn't publish its tokenizer. ellamind found that Claude's tokenizer used since Opus 4.7 needs about 65% more tokens than o200k across its 12 languages, including 59% more in English. It did not test Indian languages, so Claude's Indian-language premium is unknown.
Alibaba Qwen 3.5 (February 2026). ellamind reports that Qwen 3.5, 3.6 and 3.8 share one tokenizer with a 248,077-token vocabulary, the same size we found for Qwen3.8. If that holds, our Qwen3.8 figures apply to Qwen 3.5.
Moonshot Kimi K3. Its tokenizer file is in a format our tooling couldn't load. ellamind reports a 163,840-token vocabulary shared with Kimi K2, but published no Indian-language data.
xAI Grok 4.7. The tokenizer isn't public. ellamind measured Grok 4.5 and 4.6 through xAI's API but not in Indian languages.
Meta Muse Spark Open. Access to its files is gated. We tested Meta's open Muse Glimmer instead.
Mistral Large 4 (October 2026). We could not find its tokenizer files. ellamind reports that Mistral Small 4, Medium 3.5 and Large 3 share the 131,072-token Tekken tokenizer. We measured Small 4, which gives the same counts as Nemotron 3.5 Super.

Sources