ChatGPT BROKE the TURING TEST - New Era Begins!

Even though AI systems can mimic human speech, we need to question if they can actually think the way we do. Many top AI systems can ace tests, write essays that feel like they're from humans, and talk so naturally that it's hard to tell them apart from real people. However, when it comes to solving visual puzzles using colored blocks, they struggle. For example, GPT4, the AI that powers ChatGPT and the Bing search engine, doesn't perform well in these puzzles. And researchers actually shared these findings in May. The group that designed these puzzles hopes it will be a better way to measure AI abilities. While AI models like GPT4 can perform impressive tasks, there's a debate because they also show limitations and can't always think profoundly. Melanie Mitchell, a computer scientist, mentioned that the AI community is trying to figure out the best ways to judge these systems. Her team developed these tricky puzzles. Over the last few years, big language models have shown that they're really good at many tasks. Their main job is to predict the next word in a sentence based on huge amounts of text data they've seen when they're turned into chatbots. Humans also help fine tune their responses. What's cool is that these models can do so many things just by being trained on lots of words. While other AI systems might be better at one task, they need special training and can't switch easily between tasks. Tomer Ullman, a scientist, says that there are two main opinions on what these models are really doing inside. Some believe they show signs of true understanding. Others, including Ullman and Mitchell, are more careful in their opinions. Ullman mentions that the reason for this difference in opinion is because there's no clear proof for either side. Tests that highlight how people and AIs think differently are useful. Both sides agree these tests can show where AIs need to improve and help us better understand human intelligence. Brendan Lake, another scientist, believes that if we're going to use these AIs in important areas like healthcare or law, it's essential to know their limits. The Turing Test, introduced by alan Turing in 1950, is a famous way to check if machines can think. The test involves a human judge having a text chat with a hidden computer and another human. The goal is to see if the judge can tell who's the machine. But there's a debate on how exactly to use the Turing Test. It was more of a thought experiment rather than a real world test test. Yet over time, there were actual competitions to see if computers could pass it. These stopped in 2019. Some believe that modern AIs like GPT4 could now pass this test because they can trick many people in short chats. Experts say that if you know how these AIs work, you can usually tell them apart from humans. Francois Chalet, an engineer, suggests that one way to identify an AI is by presenting it with familiar but slightly altered scenarios. In many cases, the AI will give a reply based on its training rather than truly understanding the new situation. However, some researchers believe that using a test based on tricking people isn't the right goal. Chollei mentions that the Turing Test encourages making AI do tricks rather than useful tasks. Instead of the Turing Test, many experts prefer specific benchmarks that look at certain abilities, like language skills or math. When GPT4 was launched, its creators at OpenAI checked its abilities using machine specific benchmarks and human exams. GPT4 did really well in most of them. Mitchell mentions that while many language models score high on tests, it doesn't mean they're smarter than humans. Rather, the tests might be too easy. One concern is that these models might have seen similar questions before and they're just recalling the answer. This is called contamination. OpenAI wanted to see if this was true. They compared questions and training data. They found that even if they changed the questions a bit, the model's performance was pretty much the same. This suggests that the model isn't just recalling answers. But some Experts wonder if OpenAI's test was thorough enough. Sam Bowman, a scientist from New York University, believes that even if some of the answers come from memory, it doesn't take away from GPT4's skills. He feels the bigger picture remains impressive. However, there's a catch. Mitchell points out that these models can sometimes get a question wrong if it's worded slightly differently. For example, ChatGPT could answer a business question, but if the same question was worded a bit differently, it failed. A significant difference between humans and models is how we interpret high test scores. For humans, doing well on these tests suggests overall intelligence, the ability to handle different tasks and adjust to new situations. This isn't the case for language models. Mitchell warns that expecting them to act like humans might lead to incorrect conclusions. The way models understand language is also different. They only learn from text and don't experience the real world like humans. So while they're good with words, they might not truly understand them. Lake believes that these models show that you can be good with language without truly understanding it. But language models also have some unique skills. They can see the relationship between almost every word ever written. This lets them solve problems in their own way. Without necessarily thinking like humans. Nick Ryder from OpenAI highlights that scoring well on a test doesn't mean a model thinks like a human. OpenAI's results are only about how the model does on that specific task, not about it being similar to human thought. Other researchers have studied GPT4's abilities beyond just language. Sebastian Bubek and his team found that GPT4 could even pass tests meant to understand human thoughts and feelings. They suggest GPT4 might be an early form of a more advanced AI system. But Bubk also says that GPT4 doesn't think like a human. Mitchell compares the study to studying human cultures, which can be a bit unstructured. Ullman thinks we'd need more evidence to believe that a machine can understand human thoughts. To truly understand these models, AI experts believe we need more detailed and strict testing. They think creative logic puzzles could be a good way to do this. In 2019, before large language models became popular, Chollet created an online test for AI systems called the Abstraction and Reasoning Corpus. In this test, the AI systems are shown a series of images in which a pattern of squares changes. The AI has to understand the rule for this change and predict how the next pattern will transform. According to Chollet, the ability to adapt to unseen things is at the core of intelligence. Lake says RC shows an important aspect of human intelligence, the ability to make abstractions from everyday knowledge and use those in new, unfamiliar problems. Chollet held a contest in 2020 for bots to take the ARC test. The winning bot was trained to solve tasks similar to ARC, but didn't have any broad capabilities. It only solved 21% of the problems correctly. In contrast, humans usually solve arc problems correctly 80% of the time. Several teams have tried to test LLMs using AR, but none of them have matched human performance. Mitchell and her team came up with a new set of puzzles called Concept arc, which were inspired by ARC but had two main differences. First, Concept RC tests are easier because the team wanted to make sure they could track even small improvements in the abilities of machines. Second, the team decided on certain concepts to test and then made a series of puzzles for each concept. For example, to test the concept of sameness, one puzzle asks the solver to keep objects in the pattern that are the same shape, while another puzzle asks them to keep objects that are aligned along the same axis. The goal of this approach was to minimize the chance that an AI system could pass the test without truly understanding the concepts. The researchers then presented these concept ARC tasks to GPT4 and to 400 people they found online. The human participants scored on average 91% on all concept groups. GPT4 only scored 33% on one group and less than 30% on all the others. We showed that the machines are still not at the level of humans, mitchell said. It surprised her that GPT4 was able to solve some of the problems despite not having been trained on them. The team also tested bots from Chollet's contest, which were designed to solve visual puzzles like Arc. They performed better than GPT4, but not as well as humans. The best one scored 77% in one category but less than 60% in most. However, Bowman argues that GPT4's struggles with concept ARC do not mean it can't reason in the abstract. He points out that concept ARC is a visual test, which is not a strength of GPT4. Also, while GPT4 had to work with arrays of numbers that represented the images, human participants simply looked at the images. A version of GPT4 that can process images as input has been created by OpenAI, but it is not yet publicly available. Mitchell's team plans to test this version with concept arc, though Mitchell doesn't expect it to perform much better. Aquaviva, a scientist from the Massachusetts Institute of Technology, agrees with Mitchell. He points to a test called 1Darc, where GPT4's performance did improve, but not enough to suggest that the AI was reliably understanding the underlying rule and reasoning about it. Despite these results, Bowman argues that other experiments suggest LLMs have some ability to reason about abstract concepts. For example, an experiment with a digital version of the board game Othello showed that LLMs might be creating internal representations of the world rather than just memorizing statistics. Bowman does admit that the Reasoning abilities of LLMs are spotty and more limited than in people. However, he believes these abilities are there and seem to get better as the model size increases. He expects future LLMs to be even better. Bowman, Mitchell and others agree that finding the best way to test LLMs for abstract reasoning and other intelligence markers is still an unsolved problem. Frank, a scientist from Stanford University, does not think a single test will emerge as a successor to the Turing Test. He thinks researchers need many tests to measure the strengths and weaknesses of different systems. Wertham advises against the tendency to attribute human like intelligence to AI systems, which he calls the curse of anthropomorphization. He believes we tend to see goal oriented behavior as proof of thinking like humans, which may not be the case with AI. Put simply, even though AI systems like GPT4 might technically be able to pass the Turing Test, which is quite a feat. They still can't think or understand things exactly the way humans do. Researchers are on the hunt to find the best tests to measure their capabilities. If you enjoyed this, please hit the like button and subscribe to our channel for more content. Thanks for tuning in and catch you in the next video. Sam.