A Large-Scale Language Model for Google: The Case of Meta, a Facebook Company Trying to Build a Universal Speech Translator
Past research has shown the effectiveness of this approach, and the scale of Google’s planned model could offer substantial gains over past work. Such large-scale projects are typical of tech companies’ ambitions to dominate AI research, and draw on their advantages in terms of access to vast amounts of computing power and training data. A comparable project is Facebook parent company Meta’s ongoing attempt to build a “universal speech translator.”
Google has already begun integrating these language models into products like Google Search, while fending off criticism about the systems’ functionality. Language models have flaws like a tendency to regurgitate harmful societal biases and an inability to understand human sensitivity. After publishing papers about these issues, researchers at the search engine were fired.
Speaking to The Verge, Zoubin Ghahramani, vice president of research at Google AI, said the company believes that creating a model of this size will make it easier to bring various AI functionalities to languages that are poorly represented in online spaces and AI training datasets (also known as “low-resource languages”).
We get better performance on our low resource languages by exposing our model to multiple languages and training it in many of them. Building 1000 different models isn’t the way to get to 1,000 languages. Languages are like organisms, they’ve evolved from one another and they have certain similarities. And we can find some pretty spectacular advances in what we call zero-shot learning when we incorporate data from a new language into our 1,000 language model and get the ability to translate [what it’s learned] from a high-resource language to a low-resource language.”
To support work on the 1,000-language model it will fund collection of data for low-resource languages, including audio recordings and written texts, in order to get access to data when training across so many languages.
The company says it has no direct plans on where to apply the functionality of this model — only that it expects it will have a range of uses across Google’s products, from Google Translate to YouTube captions and more.
“One of the really interesting things about large language models and language research in general is that they can do lots and lots of different tasks,” says Ghahramani. “The same language model can turn commands for a robot into code; it can solve maths problems; it can do translation. Language models are becoming repositories of a lot of knowledge and can be used in a variety of ways.
LLaMA: An AI Large Language Model for Research Use in Artificial Intelligence (OpenAI), OpenAI, and OpenAI-Inspired AI
Rival tech companies will no doubt look at these problems and think if they launch an AI-powered search engine it’s worth it just to steal a march on Google. If you are a new player in the scene, replantational damage isn’t that much of an issue.
Openai, too, was previously cautious in developing its technology, but it was open to the public with the launch ofchatg tpe. The result has been a storm of beneficial publicity and hype for OpenAI, even as the company eats huge costs keeping the system free-to-use.
“Today we’re releasing a new state-of-the-art AI large language model called LLaMA designed to help researchers advance their work,” CEO Mark Zuckerberg said in a Facebook post. “LLMs have shown a lot of promise in generating text, having conversations, summarizing written material, and more complicated tasks like solving math theorems or predicting protein structures. Meta is committed to this open model of research and we’ll make our new model available to the AI research community.”
To this end, Meta is releasing something called LLaMA, an open source system, which is offered under a non-profit license focused on research use cases.
Meta claims that the second smallest model of the LLaMA model performs better than the GPT3 model on most benchmarking, while the largest model is competitive.
Once trained, LLaMA-13B can also run on a single data center-grade Nvidia Tesla V100 GPU. That’ll be welcome news for smaller institutions wanting to run tests on these system but doesn’t mean much for lone researchers for whom such equipment is out of reach.
Meta(e,e’m,d) Lost: AI Chatbots in the Light of Microsoft’s Launch of Bing and Google’s Stock Price
Meta’s release is also notable partly because it’s missed out on some of the buzz surrounding AI chatbots. (That might not be a bad thing, though, given the criticism Microsoft has received for rushing the launch of Bing and the nosedive taken by Google’s stock price after its own chatbot made an error in a demo.)
The reception of Meta’s own accessibleAI has been less than stellar. While one of the two software programs which was pulled offline was criticized for being simply… not very good, the other was pulled offline after only a few days because it kept producing scientific nonsense.