OpenAI Building AI Agents as Google Launches Gemini Advanced

Today on the AI Breakdown, Google has officially changed Bard to Gemini and released the most advanced version of their Gemini model. Before that on the brief, OpenAI is working towards some seriously advanced agents. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our Discord and our newsletter. Welcome back to the AI Breakdown Brief. All the AI headline news you need in around five minutes. As you might have guessed, our main story today is going to be about Google changing the Bard brand to Gemini and giving access to their most advanced ultra model which just kicked off today. However, the information has some really interesting reporting from behind the scenes in OpenAI in a piece that they titled OpenAI Shifts AI Battleground to software that operates devices and Automates tasks Now if you were an AI Breakdown listener last year, you heard me talk a lot about about AI agents. One of the huge themes among developers was trying to move to an era where instead of just answering people's questions, AI agents actually had the capacity to solve their problems. In other words, you could give them a problem and the agent could figure out what tasks were needed to solve that problem or accomplish that goal, including potentially leveraging other agents to do so. Now this is a reality that is not here yet. There are lots and lots of companies trying and experimenting and building towards that, so some of which are getting increasingly high capacity in some specific functions. But there is not yet a generalist AI agent winner or Even really leader. OpenAI appears to be determined to change that, writes the information. OpenAI is developing a form of agent software to automate complex tasks by effectively taking over a customer's device. The customer could then ask the ChatGPT agent to transfer data from a document to a spreadsheet for analysis, for instance, or to automatically fill out expense reports and enter them into into accounting software. These kinds of requests would trigger the agent to perform the clicks, cursor movements, text typing and other actions humans take as they work with different apps. Now apparently they are working on actually two different types of AI agents. The one that we just described which would take over a person's specific device and another which would be specifically for web based tasks. Now this makes sense given that Sam Altman has discussed his vision for ChatGPT as ultimately a quote, super smart personal assistant for work. However, as the information also points out, it could bring OpenAI increasingly in competition with Microsoft, who are of course positioning copilot as exactly this sort of thing. Although how far Microsoft is in any sort of plans or developments around AI agent sort of behavior isn't clear at all. There are also real questions about whether users will be comfortable with this. Right now the only types of software that take over people's computers are malware and viruses, and so getting over that impression could be really difficult. Now, these are not new efforts apparently. It appears that they've actually been in development for more than a year. However, there are some indications that employees within OpenAI think these tools are going to be a really, really big deal. One of the people who was the information sources, for example, pointed out a tweet from Ben Newhouse, who is an OpenAI employee who this source said had worked on computer using agents. And Ben on Twitter posted building what I think could be an industry defining zero to one product that leverages the latest and greatest from our upcoming models. Adding even more hype to that cryptic announcement, Pete Wellander, OpenAI's vice president of Product, added, this product that Ben was describing will quote, change everything. Now of course there are other indicators that these are certainly the lines that people are thinking about in many ways. If you go back and look at how OpenAI framed custom GPTs, it was as a very first step towards an agent like future. They of course also launched at their Dev Day event the Assistance API, which is explicitly about helping developers build light agent type experiences in their applications. So right now this is just a behind the scenes report. There's no indication that anything is coming soon, but it's consistent with other things that we've heard and it certainly seems to suggest where the AI arms race could be headed next. It is certainly something that if nothing else, I will be watching closely and letting you know if I hear anything more. Or frankly probably even less definitive. Now another company that we've been guessing at their AI strategy and who are finally starting to tease it themselves as of Tim Cook talking to investors recently is of course Apple. One of the indicators that Apple is getting deeper and deeper into its own AI strategy is the fact that they have been increasing their open source releases in the space. The latest is something called MLLM Guided Image Editing or mgie. It's a model that lets users use plain language to edit a photo without any photo editing software. So think of the sort of inpainting that got people so excited in midjourney and Adobe Image Applications last year want to be wearing a different color shirt in a photo? Just say I want my shirt to be a different color now. According to the Verge, the model blends two different uses of multimodal language prompts. First it learns how to interpret user prompts. Then it imagines what the edit would look like. Now part of what makes this different is there is reasoning involved. So for example, if you had a picture of a pepperoni pizza and you typed in the prompt make it more healthy, mgie would add vegetable toppings. This is very different than having to use a prompt add vegetable toppings. Now if you are interested in trying this out, you can download it from GitHub or you can do a web demo over on Hugging Face lastly today, another interesting survey about US Worker attitudes towards artificial intelligence. This one comes from Rutger University's Heldrick center for Workforce Development, which has multi decade experience of surveying Americans around the impact of new technology in the workplace. Now one thing that's perhaps not surprising is that there is meaningful concern among people about their job being eliminated by AI. Three in ten have that worry. However, a far more dominant worry, at least right now, has to do with the hidden hand of AI being involved in human resource decision making, I.e. hiring and firing. When it comes to those issues for AI, 7 in 10 US workers say they're very or somewhat concerned, as the Center's director Carl Van Horn puts it, a concern about the hidden hand out there that I'm not going to get a chance to really discuss my virtues with the hiring officer or with my boss. Instead there'll be some algorithm that tells me whether I stay or go. So some pretty interesting nuance we're getting into when it comes to people's perceptions and fears around AI. Always interesting to see these new stats, although of course take them as one tiny piece of evidence in a much larger world. For now though, that is going to do it for today's AI breakdown brief. I'll be back soon with the main AI breakdown. Welcome back to the AI Breakdown. Earlier this week, an Android developer found a changelog message that suggested that we would be getting the most advanced model of Gemini this week and that along with it, Google would be making a big marketing change moving the Bard brand to Gemini. Now this is something that we've seen as sort of a trend among these big companies. They start with one brand and ultimately start to settle on a cross cutting brand that refers to everything that they're touching with AI. I told the whole story of Bing shifting to Copilot yesterday, for example, which is of course encapsulated by their super bowl ad which was just released. Well, like I said at the top, the rumors are true. Bard is now Gemini and Ultra is their most advanced model is actually available. Jack Krasik from the Google Bard team says today Bard becomes Gemini available on web and mobile. New app in the Play Store starting to roll out today and introducing Gemini Advanced access to our most capable model Ultra 1.0. Jack continues, Bard was built to be the direct way to access Google's AI models. Last week Gemini Pro went worldwide and completed the transition into the Gemini era. Gemini is more than state of the art models, it's an ecosystem you will see through our products and APIs. Hence Bard is now Gemini. Gemini Advanced provides access to our most capable model Ultra 1.0. We worked with 100 AI expert trusted testers across multiple disciplines. They've told us they prefer Gemini Advanced for its longer context conversations, 32k and the ability to roleplay complex scenarios. It also doesn't interrupt your flow with low rate limits. Now from there he goes on to a lot of other details, including discussing the new app experience on Android and iOS in the Google apps, as well as another new feature called Double Check, which allows users to double check the information that's coming back from Gemini and finally discusses the pricing structure for this Most advanced ultra 1.0 model. Users can try it for two months for free and then it is $20 a month after, well $19.99 technically. Now there are some things that are not there yet that make it not as feature complete as ChatGPT. These include multimodal upgrades, interactive coding, deeper data analysis, file uploads, multilingual and more. Now there is a lot that makes this interesting, as the Verge points out. Quote Google is famous for having a million similar products with confusingly different names and seemingly nothing in common. That pattern, however, has been broken with the release of Gemini. The Verge also points out the stakes. They write it's not a surprise that Google is so all in on Gemini, but it does raise the stakes for the company's ability to compete with OpenAI, anthropic perplexity and the growing set of other powerful AI competitors on the market. In our test just after the Gemini launch last year, the Gemini powered Bard was very good, nearly on par with GPT4, but it was significantly slower. Now Google needs to prove it can keep up with the industry as it looks to both build a compelling consumer product and try to convince developers to build on Gemini and not with OpenAI. Only a few times in Google's history has it seemed like the entire company was betting on a single thing once that turned into Google, and we know how that went. But this time it appears Google is fully committed to being an AI company, and that means Gemini might be just as big as Google now. The only thing that I disagree with from this Verge analysis is the idea that going all in on Gemini raises the stakes for the company's ability to compete. Those stakes were raised by the very existence of leaders in the AI space that weren't Google. For a company that has been at the very forefront of innovation in this space, it was shocking last year to see how far it was behind throughout the entire year. It was, in fact, to many people, stunning indeed. It put them in a position where they really had to announce Gemini in December, even though they couldn't make Ultra available at that time. The exciting thing is, of course, is that because we didn't have access to the Ultra version of the model back in December, we simply had to take their assertion that it matched or exceeded GPT 4 in numerous areas at face value, or choose not to believe it. Now we get to test it for ourselves, but there are some people who have had a little bit longer of a chance to already dig into it. Popular creator Marques Brownlee says okay, so I've been testing out Google's Gemini on a few phones for a few weeks now. Some things I've noticed that stood out Tons of useful new generative features can write letters, craft trip plans, create images, etc. All the good stuff. Notably better semantic understanding of random fact phase questions. Downside, it's missing some classic Google Assistant features like home control and adding to shopping lists. The new popup UI is a little more complex, but you can get used to it. Renaming it is going to confuse a lot of people for no reason. Bindu Ready from Abacus writes my initial thoughts still continues to be somewhat nerfed and refuses to answer questions Refuse to generate a simple illustration of George Clooney. ChatGPT is better. Missing PDF upload answers do seem better than the previous version. Seems to have a reasoning vibe. However, it does not answer some hard questions that GPT does. For example, it didn't get in a room. I have only three sisters. Anna is reading a book. Alice is playing a match of chess. What's the third sister, Amanda doing? The answer is the third sister is playing chess. GPT4 nails it Overall, we plan to do a lot more analysis, but first impressions are good but not great. Tldr I don't think it will make a material difference to how Bard was doing before, especially if their plan is to charge for this. However, it's always good to have more players in the market. Now someone who has a much more positive take is V. Mousowicz, who writes, as someone who had early access, I can say that Gemini Ultra is damn impressive. When it is good, it is excellent and this includes the most common queries, especially learning and looking up facts. I've switched to it as my default LLM. Now Zvi is a prolific blogger and super compelling thinker, so I am inclined to take his take on this with a little bit more confidence than some of the others. And then of course there's Professor Ethan Mollock from Wharton. Ethan has had access to Gemini for the last six weeks and wrote an extensive post on his One Useful Thing blog giving his perspective. The post is called Google's Gemini Tasting Notes and Implications subtitle and then there were two now one thing that Ethan makes clear is that he is not trying to test Gemini on the basis of benchmarks. Instead he wanted to give a subjective mix of opinions based on his usage. He writes Let me start with the headline Gemini Advanced is clearly a GPT4 class model. The statistics show this, but so does a month of our informal testing. And this is a big deal because OpenAI's GPT4, the paid version of ChatGPT and Microsoft Copilot, has been the dominant AI for well over a year and and no other model has come particularly close. Prior to Gemini, we had only one advanced AI model to look at and it is hard drawing conclusions with a data set of one. Now there are two, and we can learn a few things. Now I just want to stop here and put a fine point on this. It really is remarkable that for an entire year in this insanely fast moving space, there was nothing that could really come close to GPT4. I've actually argued in the past that having another GPT4 class model on the scene and actually available represents a major transitional moment for the industry from the period that was entirely defined by ChatGPT from November 2022 when it launched to basically today to this new era that's coming, whatever it happens to look like now. That said, Ethan continues that Gemini Advanced does not obviously blow away GPT4. He writes, it is really good, but I would concur with the tests that suggest it is roughly equivalent when it comes to the various strengths and weaknesses of these platforms. He writes, GPT4 is much more sophisticated about using code and accomplishes a number of hard verbal tasks better. Gemini is better at explanations and does a great job integrating images and search. Both are weird and inconsistent and hallucinate more than you would like. And that gets to a really interesting section of the piece called It's Full of Ghosts. Ethan writes, no one has a great definition of sentience, which is okay because LLMs are in no way sentient. They are software systems designed to create human like language. But there is a weirdness to GPT4 that isn't sentience but also isn't like talking to a program. A weirdness that only comes out after you spend enough hours playing with the AI and getting unnerved or delighted or both by its unexpected abilities and seeming intelligence. There was a famous controversial paper put out by Microsoft Research soon after the release of GPT4 called Sparks of Artificial General Intelligence that tried to put this argument into scientific terms, but ended up just calling it Sparks of Artificial General Intelligence. It is the illusion of a person on the other end of the line, even though there is nobody there. GPT4 is full of ghosts. Gemini is also full of ghosts. Seriously, if you use the system for a while, I can almost guarantee at least one moment when you stand up from your desk, walk around the room and wonder what is going on. Now, his takeaway is that the sparks that we saw in GPT4 are not based on GPT4, but are byproducts of GPT4 class models. From a tone and personality perspective, he suggests that GPT4 is more bland, where Gemini is more friendly, agreeable and has a, quote, tendency towards wordplay. But ultimately he says, these models are really, really similar. Now the other thing that he argues about Gemini is that it illuminates a vision of AI as a powerful integrated personal assistant. He basically argues that the barred integration with the Google ecosystem of Gmail, Google Docs, travel tools, et cetera was interesting but too dumb to actually use. Whereas now he says with a smarter brain in the form of Gemini Advanced, you can start to do some really interesting things that at their best seem magical. Go through my emails, tell me which are important and draft replies for each. Let's up my next conference and plan a trip I would like. He says it isn't there yet, but it is very much closer to being an actual assistant rather than the limited series at Alexis we have seen in the past. That is in part why I suspect that Gemini Advanced is the start, not the end of a wave of AI development. We can start to see a world where AI agents act on our behalf. A GPT4 class model is not quite strong enough to power these agents, but we are getting close. So really interesting stuff and ultimately what we come back to is that we are now living in a 2 GPT 4 level world, which I think is going to have some fairly significant implications that we will discover as it happens. For now though, an exciting day in the AI world. I appreciate you listening or watching as always. And until next time, peace.