A research has revealed that people who talk to AI tools in other languages instead of English, AI has to spend more tokens to process their conversations. This makes AI tools expensive to use. The researcher says that there is a strong need for AI companies to train their models in other languages.
How did this problem of AI come to light?
According to the report, recently Aran Komatsuzaki, a researcher at OpenAi, conducted an experiment. They explored how OpenAi and Anthropic’s systems process different languages. Aran got an article translated into many languages by Ai.
- After translation, the researchers found out how many tokens the AI spent on the task.
- A large difference was found between tokens spent in English and other languages.
- OpenAi’s AI model required 1.37 times more tokens to understand Hindi text than English.
- It took more effort for Claude AI to understand Hindi. This difference increased to 3.24 times.
- Similarly, 2.86 times more tokens were spent for Arabic language and 1.71 times more tokens were spent for Chinese language.
- According to the report, if an English speaking user spends 100 tokens to convey his point, a Hindi user will have to spend more than 300 tokens to explain the same thing on Cloud AI. (ref.)
Why does AI spend more tokens?
- According to the report, the reason for this is that the AI model breaks any text into small pieces.
- Any model breaks the written sentences into smaller parts to understand what its user is saying, which is called tokenizer.
- All AI tools have been trained in English. AI systems process English easily.
- They have to spend more tokens in breaking Hindi and other languages, because the handwriting of these languages is different.
- However, this does not mean at all that AI has any problem in understanding Hindi.
When will AI be able to solve this problem?
According to reports, there is no timeline as to when this problem will be resolved. Nor has any solution been found for this.
Researchers say that AI companies will have to pay attention to the training of their AI models. Such systems will have to be prepared which can handle other languages.
About the authorPrem TripathiPrem Tripathi is Assistant Editor at Times Online. He is leading the Tech-Auto team. He has been in journalism for 15 years covering the fields of technology, gadgets, artificial intelligence and defense tech. Prem has seen India’s smartphone market change from the dominance of Nokia to Samsung and Chinese companies to the growing popularity of Apple. He has seen that era of telecom journalism when 2GB data available on 2G and 3G mobile networks had to be used for the whole month. This is the reason why when readers have to give news about research being done on 5G network or 6G, their experience is clearly reflected in the news of love. He knows how to explain topics like ‘Satellite Internet’ and ‘Direct to Device Communication Technology’ in easy language. It’s been a long time since Prem has been writing gadget reviews. He has done in-depth reviews of everything from early Windows to Android smartphones and iPhones. From laptop-PC to smart TV and wearables have also been thoroughly tested and tested. Prem does detailed interviews of experts from the tech-telecom industry. He has been covering launch events on ground for many years. Covering the India Mobile Congress (IMC) since its first edition (2017), he has gained expertise in understanding the major changes happening in India’s technology industry. In recent years, Prem has strengthened his writings on the topic of Artificial Intelligence (AI). He uses AI tools and tells readers about them. Every major change taking place in the field of AI is explained to the readers in easy language. With the idea of how technology can make people’s lives easier, Prem also provides simple HowTo, Tips & Tricks to the readers of Times. He first tries and verifies any hack himself. He does the same practice with daily tech-gadget news also. By verifying news, cross checking facts, gathering additional inputs, research and adding statistics, he prepares any news in detail for his readers. Prem has great experience in print media and digital media. Before joining Times Online, he was working with the reputed tech-gadget website, Gadgets360 (NDTV Ventures) from November 2021 to January 2025. There he provided detailed coverage on gadgets, tech, AI, defense tech and science and wrote gadget reviews. Prem started his career with Hindustan Hindi in the year 2011. Served in Amar Ujala newspaper in 2013 and Times newspaper in 2017. He has worked in both reporting and desk. Covered ‘Kedarnath Disaster 2013’ for Amar Ujala. His book ‘Hai Gau’ was published by Amar Ujala Foundation. Prem Tripathi completed his Bachelor of Science (Bsc) from Kumaon University, Nainital in 2008. He holds a Master of Arts (MA) degree in Mass Communication from Makhanlal Chaturvedi University, Bhopal.… read more