AlphaProof and AlphaGeometry: a “Strawberry” for Artificial Intelligence to Clearly Explain Mathematical Questions
The technology underlying most progress in Artificial Intelligence of late, with the language of traditional programming, is what underlies the approach used by AlphaProof and Alphageometry.
Researchers have achieved mixed results when trying to answer mathematical questions with language models — the type of system that powers chatbots such as ChatGPT. Sometimes, the models give the correct answer but are not able to rationally explain their reasoning, and sometimes they spew out nonsense.
In the future, the neural-symbolic method could provide a means for AI systems to turn questions or tasks into a form that can be reasoned over in a way that produces reliable results. It is rumored that Openai is working on a system called “Strawberry.”
There is, however, a key limitation with the systems revealed today, as Silver acknowledges. AlphaProof and AlphaGeometry are able to work towards the correct answer with correct or incorrect math solutions. Many real-world problems are not easy to solve, such as figuring out an ideal itinerary for a trip. The solution for more ambiguous questions could be to use a language model to try to determine what is a “right” answer. “There’s a spectrum of different things that can be tried,” he says.
DeepMind and other companies are in a race to eventually have machines give proofs that would solve substantial research questions in maths. The problems set at the world’s premier competition for young mathematicians have become a benchmark for advancement towards that goal, and have become seen as a grand challenge for machine learning.
Their new Frankensteinian creation, dubbed AlphaProof, has demonstrated its prowess by tackling several problems from the 2024 International Math Olympiad (IMO), a prestigious competition for high school students.
In some cases, AlphaProof provided a correct step out of a large range of possibilities. It would take further analysis to find out if the answers were less surprising than anticipated. A similar debate ensued following the surprising ‘move 37’ taken by DeepMind’s AlphaGo bot in its famed 2016 defeat of the world’s top human Go player — a watershed for AI.
“This is the first time any AI system has been able to achieve medal-level performance”, said Pushmeet Kohli, vice-president of AI for science at DeepMind, in a briefing to reporters. “This is a key milestone in the journey of building advanced theorem provers,” said Kohli.
The researchers utilized Alphageometry2 to solve the geometry problem in less than 20 seconds, and the improved and faster version of their record-setting system, according to a DeepMind computer scientist.
Numina, HuggingFace, and AlphaProof: a case study of open and closed problems in the Iomimium Open to all
Just last week, a team of researchers from software companies Numina and HuggingFace used a language model to win an intermediate AIMO ‘progress prize’ based on simplified versions of IMO problems. The companies made their entire systems open-source and available for other researchers to download. But the winners told Nature that to solve harder problems, language models alone would probably not be enough.
AlphaProof could start its reinforcement-learning cycles if enough good translations from Lean were not nonsensical. The results were better than expected, according to the press briefing. “Many of the problems in the IMO have this magic-key property. The problem looks hard at first until you find a magic key that unlocks it,” said Gowers, who is at the Collège de France in Paris.
“It’s clearly a very substantial advance,” says Joseph Myers, a mathematician based in Cambridge, UK, who — together with Fields Medal-winner Tim Gowers — vetted the solutions and who had helped select the original problems for this year’s IMO.
“We’re at the point where they can prove not open research problems, but at least problems that are very challenging to the very best young mathematicians in the world,” said DeepMind computer scientist David Silver, who in the mid-2010s was the leading researcher in developing AlphaGo.
Whether the techniques can be perfected to the point of doing research-level work in mathematics remains to be seen, Myers said in the press briefing. “Can it extend to other sorts of mathematics where there might not be a million problems to train on?”