How to Build a 1,000-Language Language Model for Research: Google, Google AI, and Isometric Modeling of Languages
Dean said that they are looking to get these out into real products and into things that are more prominently featuring the language model rather than under the covers. It is important that we get this right. Pichai added that Google has a “a lot” planned for AI language features in 2023, and that “this is an area where we need to be bold and responsible so we have to balance that.”
While fending off criticisms about the system’s function, the company has begun integrating the language models into products. Language models can have a number of flaws, such as a tendency to put biases in their language and not understand human sensitivity. Google itself infamously fired its own researchers after they published papers outlining these problems.
Speaking to The Verge, Zoubin Ghahramani, vice president of research at Google AI, said the company believes that creating a model of this size will make it easier to bring various AI functionalities to languages that are poorly represented in online spaces and AI training datasets (also known as “low-resource languages”).
“By having a single model that is exposed to and trained on many different languages, we get much better performance on our low resource languages,” says Ghahramani. “The way we get to 1,000 languages is not by building 1,000 different models. Languages are like organisms, they’ve evolved from one another and they have certain similarities. And we can find some pretty spectacular advances in what we call zero-shot learning when we incorporate data from a new language into our 1,000 language model and get the ability to translate [what it’s learned] from a high-resource language to a low-resource language.”
Access to data is a problem when training across so many languages, though, and Google says that in order to support work on the 1,000-language model it will be funding the collection of data for low-resource languages, including audio recordings and written texts.
The company says it has no direct plans on where to apply the functionality of this model — only that it expects it will have a range of uses across Google’s products, from Google Translate to YouTube captions and more.
“One of the really interesting things about large language models and language research in general is that they can do lots and lots of different tasks,” says Ghahramani. The same language model can do things such as solve maths problems and turn commands for a robot into code. Language models are becoming a repository of a large amount of knowledge, and by probing them in different ways you can get to different parts of useful function.
After the launch ofchatg.pitt, there are new discussions about the potential of chatbot to replace traditional search engines, although the issue has been under consideration for a long time The same challenges that Pichai and Dean are explaining to staff, are what caused Timnit Gebru and Margaret Mitchell to be fired from the company. And in May last year, a quartet of Google researchers explored the same question of AI in search, and detailed numerous potential problems. As the researchers noted in their paper, one of the biggest issues is that LLMs “do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over.”
Additionally, the creators of such models confess to the difficulty of addressing inappropriate responses that “do not accurately reflect the contents of authoritative external sources”. A text on the benefits of eating crushed glass and a paper on how crushed porcelain adds to breast milk are just two examples of what can be created. In fact, Stack Overflow had to temporarily ban the use of ChatGPT- generated answers as it became evident that the LLM generates convincingly wrong answers to coding questions.
Yet, in response to this work, there are ongoing asymmetries of blame and praise. Model builders and tech evangelists alike attribute impressive and seemingly flawless output to a mythically autonomous model, a technological marvel. The decision-making that takes place in model development is erased and model feats are observed as independent of the design and implementation choices of its engineers. It becomes harder to acknowledge the responsibilities when you don’t know what the engineering choices are that contribute to the models. As a result, both functional failures and discriminatory outcomes are also framed as devoid of engineering choices – blamed on society at large or supposedly “naturally occurring” datasets, factors those developing these models will claim they have little control over. But it’s undeniable they do have control, and that none of the models we are seeing now are inevitable. It was possible for different choices to be made, resulting in a different model being developed and released.
AI Language Models: OpenAI’s Launch of ChatgPft and its Impact on Search Engine Optimization and a Comparison to Google’s AI Test Kitchen
Google has developed a number of large AI language models (LLMs) equal in capability to OpenAI’s ChatGPT. These include BERT, MUM, and LaMDA, all of which have been used to improve Google’s search engine. The focus of such improvements is to improve their understanding of their intent. Google says MUM helps it understand when a search suggests a user is going through a personal crisis, for example, and directs these individuals to helplines and information from groups like the Samaritans. Google has also launched apps like AI Test Kitchen to give users a taste of its AI chatbot technology, but has constrained interactions with users in a number of ways.
Openai was cautious in the development of its technology, but has changed tack with the launch of the chatgpft. The result has been a storm of beneficial publicity and hype for OpenAI, even as the company eats huge costs keeping the system free-to-use.