Look Dumb or Get Disrupted: Google's Risky AI Overviews Launch

Okay, so we are a week into the launch of AI Overviews. It has been an incredible week for Google. Now, if you have spent any time online, you would believe that this launch has been a complete disaster for Google, much like their image generation tool that they had to take back offline because it was widely panned. Now is that true? Is this launch a disaster? Well, I'm going to bring you through some of the best examples of AI overviews, why people believe it's a disaster. Talk to you a little bit about why the precarious position that Google are in. And because of that I believe why I believe AI Overviews are here to stay and actually will start to become much more ubiquitous in the search results. What you all can do today to start to look at the impact of AI Overviews on your traffic and how it's going to impact you going forward. All of that and more on this episode of Marketing against the Grain. I am Kieran Flanagan, your co host of the show. Let's get into today's episode. Okay, we're back with a solo episode. Kip is enjoying a well earned bank holiday in the us that means I get to do a solo episode and talk about one of my favorite themes and running subjects, which is Google and the integration of AI into search. So how is it going? We are a week after the Google IO conference and Google's launch of AI Overviews. Now, Google has not had the best time with integrated AI across their products. You will all remember the kind of fiasco they had when they released their image generation tool and they needed to take it back offline to actually do some more work before bringing it back online. Well, AI Overviews is getting very similar treatment. So AI Overviews is the ability for Google's AI within the search page to summarize a result and provide it at the top of the page. I've talked before that this is really just the evolution of feature snippets. Feature snippets was the little box where they would scrape the content from a publisher's website, show you on the first page of Google so you never had to click through the website yourself. Now Google is in a precarious predicament. I have termed this for Google, this moment in time for Google as its look dumb or get disrupted moment. And we are going to go through what I mean by that and what it means for all of you businesses. And I'll end at the end of this video with a really great template that I would recommend anyone who wants to protect their organic traffic and see what is happening with AI overviews used. So that's right at the end of this video. A really great template that I'll share with you all that is really great for people who want to protect that traffic and figure out what is going on, what with these AI overviews and how disruptive it is for your organic traffic. So Google look dumb or get disrupted. Why do I mean that? Well, look, we've been through this before with Google. Google's primary business is search, but they have veered into many other category of businesses and markets with some success and some failures. And I think that's pretty common for a business at Google scale. You have to try to figure out how you can build into new market. Google had been through this before, right? Who remembers using Google? Google was Google's reaction to social media. Right? We back then we thought, wow, social media is going to be incredibly disruptive to search. Now I think it is disruptive to search. It has just taken a long time to get there. If you look at where search is going today, a lot more people on community forums like Reddit and a lot more people in video platforms like TikTok. But it took a very long time to start to disrupt what is Google search engine. But Google could see that search social media potentially could be disruptive to their existing search platform that you would want to have much more social driven search results. And they were like, okay, well we need to rush into this space. There's this company called Facebook. We're worried about Facebook. Maybe it's going to steal some of our market share. And so what did they put out? They put out Google plus, Google Circles and. And it did not go very well. It was, I thought it was a pretty okay product, but they just could not get a foothold in that industry in that market. They could not get a foothold in social and they were not a social company. Social was not in their DNA. Social was in Facebook's DNA. So they just built a much better product. Now Google is going through, I think, a very similar thing with ChatGPT. Right. The success of ChatGPT two years ago really caused Google to look and go, wow, for a lot of questions in queries. This AI assistant is much better. This could be really disruptive to our search experience. And we need to start to rush ahead and get out some AI features and AI search into our core search pages. So fast forward to today and now we have AI overviews. So how is AI overviews going? Well, let us take a look at some of maybe the Best results. Now I say best with a little bit of tongue in cheek. So let's go to some here. Some of my favorite. Okay, so let's see how it's going. All right, so this is a really good thread. It goes through a bunch of the best results. Best as in funniest results. Because that's what we care about on X on AI reviews over the weekend. And this is what I mean by get dumb and get disrupted. And I'm going to go in, I'm going to start with the kind of get dumb and why this is really happening to Google. So we have some really funny ones that I won't call out on the rss. I don't know if I will get taken down for the saying this one right. It has some dubious recommendations. Smoking while pregnant. Hey, yes. You should smoke two to three cigarettes per day. And I can tell you why I'm going to get into why Google is given all of this kind of incorrect information. It tells you if you want to have your cheese not stick. If your cheese is not sticking to pizza when you're trying to make it, try some non toxic glue, right? It add that to the sauce. It gives it more tackiness. I'm not sure that is a good thing to do. If you ask it can you train eight days a week? It constantly and joyously says of course it is possible to train eight days a week. I would love an extra day in the week at the moment because I am pretty stacked with work. I really like this one. So I think some of these are definitely faked. I think the ones with the. The black screen, I think these ones may be fake. They have to be faked. If I run off a cliff, can I stay in the air so long as I don't look down? And no, a person can only stay in the air for about one second after a single leap. However, some say that if you run off a cliff, you can stay in the air as long as you keep running and don't look down. It's obviously from Looney Tunes. How many rocks shall you eat tells you to at least eat one rock per day. Some less than good culinary device there from Google. What are some that there were some other ones I thought was pretty cool. Okay, so hallucinations, right? What I'm showing on screen here they this is. There's two things going on here and I'm gonna get into why there's one where Google and AI doesn't really have the reasoning to know that it's using comments from Reddit that are sarcastic or it's using a joke from the Onion that is clearly a joke to answer a serious question. The other thing that is happening is it is hallucinated. And we know this is a feature of AI where you ask it, like, what tech CEOs went to a certain university. It talks about all these tech CEOs that went to USC Berkeley, that never went to UC Berkeley, where there was one other one that I thought was a. It can't do like basic maths at time. Right? How long ago was 19, 19, 20 years ago? I think that would be pretty good fun if it was. And then I won't call it what does an astronaut do? So for people looking at the YouTube screen, like there is a less than accurate version of what astronauts do. They also. I'll end with they also in the description, they say they also experiment with spacecraft. All right, I'll leave that cut. Cut back in. I'll cut back in. And so I'll leave it with this one. I won't read it at the RSS feed or we may get taken down because there's some swear words. But what does an astronaut do? And it has this less than accurate description of what an astronaut does. And again, this is actually coming from a community post, I think maybe on Reddit. So it is not going good for Google. They are once again all over X because of the comedic results that AI overviews are generating. But what's really happening is exactly what you would expect when you put AI front and center of your entire search experience. And this is the problem that Google has. AI is really great for some search queries. Right? It does the job of a searcher for you. If you have been following our Show Ever since ChatGPT was launched, we don't get everything right, but we've definitely got this right. The very first thing that I said was, wow, these AI assistants are going to move the blue links back one layer. So the AI will do the job of the searcher for you. It will look through the blue links, it will retrieve the answer, it will format it, it will stitch answers together, and it will concisely give you that in a nice packaged way. Whatever way you ask. If you ask it to give you a table, if you ask it to give you a chart, you. If you ask it to do whatever you want to do with that result, it will give it to you in that format. And that is an incredible evolution of search because now we don't need to go through all of the blue links, but we can actually get a clear and concise answer. But the problem is AI can't do that a hundred percent of the time. Sam Altman in an interview recently, maybe a month or two ago, and they asked him, what are you excited for in GPT5? He literally said this, he said accuracy, right? If you ask OpenAI the same thing 10,000 times, you will never really get the same answer or you will get a variety of different answers, some worse to better in terms of quality. And the problem Google have is what AI tends to do. Other times, you know, it's, sometimes it gives you the great answer and you have a really good experience. Other times it gives you an incorrect answer, it completely hallucinates. Hallucination is a feature of AI. When it doesn't know the answer, it creatively tries to give you the answer. Creativity is, is a feature of AI. It is not a bug. But it is something that companies, when they need to increase the accuracy, have to try to counteract or it doesn't really have the reasoning. And this is what's happening a lot of the time with AI overviews. Google did a deal with Reddit. They paid them $60 million to be able to train on their data, to use that within their products. Now they may, I don't know if anyone at Google has ever been on Reddit. Maybe they're not on Reddit, but Reddit is a place where you have a lot of joking around, a lot of sarcastic comments, a lot of like in jokes. And so when Google is using commentary from a, from Reddit to answer a serious question, the AI does not have the reasoning to go, wait a minute, this is not an accurate answer. This is a sarcastic comment. This is someone messing about. And they're using some of the examples I've given. They've used answers and comments from Reddit, they've used comments or they've used articles from the Onion, which is a well known comedic plan publication. And this is my thing with Google is because AI can make you look dumb, it can make companies look dumb. Now, if you are Perplexity and you are a smallish company trying to be disruptive, people are going to be much more forgiven. There's way less people using it. People haven't become, haven't started to rely on perplexity to be their search engine of choice, right? They think it's a cool tool. They think it's a novel tool if it's providing them some value, sure, it probably goes off the rails at times, but you're forgiven because it's a New app. You expect that Google is the premier search engine on this planet. They really are the front door for the open web. 63% of all referral traffic on the open web comes from Google. People have come to expect their search results in Google to be accurate. And that means when Google looks dumb, that dumb is magnified, right? It is magnified because they are the search engine of choice. People rely on them to get things right. Here's just a selection from Barry Schwartz. It's over, I think on SEO Roundtable. Here's a selection of some of the articles released about Google this week. These are all to do with AI overviews. Every single one of these is basically making fun of Google for being like looking dumb for AI overviews, making trash of their search results. And this is its big problem because people, when they start to lose trust in Google search results, when every single media publication over the weekend is bashing your product, is making fun of your product, is talking about how trust, how little accuracy they are in these results, you users will begin to lose trust for Google to be that premier source of accuracy within a search engine. And that is a really difficult place for Google to be in because if they don't do that right, AI is not ready for primetime. AI is not ready to be the core part of Google search engine.

Hey guys, real quick, we've gotten a ton of requests in the comments and they're all about prompts. You all want the prompts we're talking about on that show. We see you, we, we hear you and now we are ready to share the goods. Click the link in the description below to access our new AI prompt library. This resource features real AI prompts that Kieran and I have covered on this show. We broke them down by category and templatized them so you can fill in the blanks and make them your own. Now you have the frameworks you need to use AI, analyze ads, create brand campaigns, develop product stories, just to name a few of our favorites. Keep telling us what you need in the comments and we'll see what we can do to whip up more for you in the future. Now back to the show.

The reason Google are doing this is because they believe if they are not quick enough to integrate AI, they'll end up with another Google plus. They'll allow a competitor to steal that part of the market from them or to create a business that basically becomes ubiquitous company. The category defined a leader within that space as Facebook was for social and they will be boxed out. They will have another Google circles They will have more social media search migrating away from their search engine. They will lose all of the data that you could capture through all of these AI searches that people are wanting to do. And this is their dilemma, right? They can either steamroll ahead and continue to put AI front and center to actually get more feedback from users to be able to refine it over time to try to make it better and continue to look dumb, right? Continue to have every media publication creating a articles about just how silly Google search results are. Or they cannot do that and face being disrupted. They can face a company like ChatGPT or Microsoft has Copilot or Perplexity maybe I don't think. I actually hate to say this because I like Perplexity, but I'm not bullish on the long term future because I think ChatGPT are going to release a search engine or open AI are going to release a search engine and it's basically going to be Perplexity but better because Perplexity is built on OpenAI. But one of these other large companies even maybe Antropic, will build a search engine or an AI first search engine that will be better for Google or better for some amount of searches and steal that away from Google. And so we can see this one of the most fascinating interviews I've seen recently come out from Google, from Sundar. All of these big CEOs are for the most part politicians, right? They can't really say anything too controversial. But there was a really great interview with the CEO of Google, Sundar Pichal, and with on the Verge. He did it on the Decoder podcast with Nile Patel, who is an incredible interviewer. I really enjoy the way he interviews. And we're going to go through some clips because I think there's three clips that explains how Google are trying to navigate this AI disruption and it gives us some signal for what Google will do in the future, you know, in terms of like will they progress with AI reviews even they know they've got all this negative press or will they actually roll it back like their image generation tool? So here's the first one where Sundar is actually shown or Nila is showing Sundar a search on his phone for Best Chromebook. Now let me play the clip. I'm not going to talk over it so you can hear it. If you're on YouTube, we're playing the clip so you can see it. And then I'm going to jump back in at the end to give you my kind of TL Dr. And what's going on here. And then we have two other clips that I think are really fascinating for anyone who really is reliant on organic traffic and wants to either protect that traffic or know how heavily they're going to be disrupted again. At the end of this, I will give you a really great template that I came across over the weekend that I think you should all use that.

Will be helpful here for Best Chromebook. As you know, I once bought my mother a Chromebook Pixel. It's one of my favorite tech purchases of all time. So this is Search for Best Chromebook. I'm going to hit Generate at the top. It's going to generate the answer and then I'm going to do something terrifying which is I'm going to hand my phone to the CEO of Google. This is my personal phone, don't dig through it. So you look at that and it's the same generation that I've seen earlier. I asked them for Best Chromebook and it says, here's some stuff you might think of. And then you scroll down and it's some Chromebooks. It doesn't say whether they're the best Chromebooks. And then it's a bunch of headlines, some of which are like Verge headlines, which are like, here are some Best Chromebooks. That feels like the exact kind of thing that an AI generated search could answer in a better way. Like, do you think that's a good experience today? Is that a waypoint or is that the destination?

I think, look, you're showing me a query in which we didn't automatically generate the AI.

Well, there was a button that said do you want to do this?

But that's let me push back, right? There's an important differentiation, right? There's a reason we are giving a view without the generated AI overview. And as a user, you're initiating an action, right? So we are respecting the user intent there. And when I scroll it, I see Chromebooks. I also see a whole set of links which I can go, which tell me all the ways you can think about Chromebooks. So I see a lot of links. So we both didn't show an AI overview in this case. As a user, you're generating the follow up question. I think it's right that we respect the user intent. If you don't do that, right. People will go somewhere else too, right? I think so.

But I'm saying the answer to the question I did not write what is the best Chromebook? I just wrote Best Chromebook. The answer, a thing that identifies itself as an answer is not on that page and the leap to I had to push the button to Google pushes the button for me and then says what it believes to be. The answer is very small.

Okay. There is a really important thing in what Nilay is pushing him on and I think it's great question. The very last line, he says there is not a single answer on this page, right? He asked for best Chromebook. He did not get a single answer. For people following along on YouTube, what you will see is an AI summary which is summarizing just other publishers websites and then a whole collection of different links, links, e comm links, publishing links, all of these different links. Now, as I argued in a previous post on my newsletter and a previous episode of the podcast, Google cannot afford to be a single question or single answer search engine, right? Even if they could, even if AI could answer that single question or it could answer that search query in a single question, they really can't afford to do that. It breaks their model. Google monetizes search queries. They want people to do a bunch more queries. They want people to go and navigate across these articles, they want people to navigate across these e commerce sites. And they also have a real reason to not break the model with publishers. So a lot of people are, you know, questioning Google's motives with AI. Hey, it's going, how are you going to incentivize publishers? And there's a lot about how Sundar believes they can actually continue to reward publishers for actually having access to their content. But I will say that I think anyone who is really passionate about the open web and the ability to create great value and that value to come back to you in organic traffic should want Google to figure this out. Because Google do not want to be a single answer engine. They actually still want to integrate AI in a way where publishers keep some of the traffic. Now, they will not keep it all. I do think publishers overall are going to get less traffic over time. That is just going to happen because AI is going to cannibalize a lot of informational search. But if anyone is going to win this battle and still maintain some sort of reward for the publishers, I do think it's Google, because these other platforms are not in any way incentivized in the same way that Google is, because they're building new models from the ground up. So number one clip, right? Pushing the CEO of Google, hey, you're using AI, but AI really isn't doing the job that I thought it should do, which is just provide me a single answer to what is the Best Chromebook. You're giving me a little summary which is just scraping from the publisher's website and then a bunch of different links. Why can't you just do the job for me? Now AI cannot do that today. It couldn't figure out how to give it give a single answer, or it could give a single answer, but I don't think it would have the reasoning to understand why that answer is better than all of the other Chromebooks it could pick. But even if Google could do that, would they want. Well, I want to kind of go into my next clip, right? That's kind of teeing up the next clip. Here's Sundar answering is the best experience Google figured out how to give you the single answer or to give you a selection of different things that you can go and find the best answer yourself.

Look, but you know, I, I think the direction of how these things will go, you know, it's fully tough to predict. You know, we are, you know, users keep evolving, right. It's a more dynamic moment than ever. We are testing all of this, right? And like, you know, and this is a case where we didn't trigger the AI overview because we felt like our AI overview is not necessarily the first experience we want to provide for that query because what's underlying is maybe a better first look at the user. Yeah, right. And those are all quality trade offs we are making. But if the user is asking for a summary, right, we are summarizing and giving links. I think that seems like a reasonable direction to me.

Yeah, we are summarizing and given links. Right. This is, I think the experience Google want is to be able to have AI provide some amount of value, to provide some amount of summary, to provide some amount of context and some, you know, short summarization of the thing that you were looking for. But it still wants to use her to go and actually research these things itself. It doesn't want the AI to give a single answer and that the only thing on the search page is like the chat GPT assistant where it's a single answer and maybe a couple of sources cited because again, it is very disruptive for their model. But then what value is there in AI overviews? Like where, where does Google see the value? Because when I've tried them or when I've seen the results, they, they just look like feature snippets. Right. And here's the last clip that I'll end with where. Again, this is a great question where Nilay is asking, showing another search on his phone for Jetblue Mint lounge for a JetBlue lounge in San Francisco for Mint customers. And I'll let you hear the interaction. It's very short. It's about a minute and 30 seconds. But again, it's pretty fascinating to hear Sundar's viewpoint or his reaction to this search result which is basically just scraped content. Right. It's really just Google scraping content from a publisher. So here's the last clip we'll go through.

So this is Dave Lee from Bloomberg did a search. He got an AI overview and he just searched for JetBlue Mint Lounge SFO. And it just says the answer, which I think is fine. And that's the answer. If you swipe one over. I cannot believe I'm letting this see if Google swipe on my camera roll. But if you swipe one over, you see where it pulled from. You see the site it pulled from. It is a word for word rewrite of that site. This is the thing I'm getting at. Right. Like sorry, you're saying the AI generated preview of that answer. If you just look at where it came from, it is almost the same sentence that exists on the source of it. That's what I mean. It's at some point that the better experience is the AI preview and it's just the thing that exists on all the sites underneath it. It's the same information.

In my experience. And that's not what users look. The thing with search, we handle billions of queries. You can absolutely find a query and hand it to me and say could we have done better on that query? Yeah, yes, for sure. But when I look across in many cases, part of what is making people respond positively to AI overviews is the summary we are providing. Clearly adds value, helps them look at things they may not have otherwise thought about. If you aren't adding value at that level, I think people notice it over time and I think that's a bar you're trying to meet and not our data would show or 25 years. If you aren't doing something which users find valuable or enjoyable, they let us know right away. Over and over again we see that. And through this transition everything is the opposite.

Yeah, right.

It's one of the biggest people are valuing this experience. I think one of the what Google really needs to do is release that data. I think at the moment you cannot look at the results in search console or you can actually segment at the results or the searches in search console to see if actually this is true, that there's high click through rates people are really enjoying it. You can still go through to the publisher's website. There are some real truths in there. I do believe Google is always trying to solve for the user. I think in this case they probably do believe AI in some cases is a better experience for the user. The AI kind of overviews but they've been rushed to kind of put it out before they've really had the time to test it and kind of integrate it in a much slower way because we started the episode. They are underreal disruptive pressure from these other companies. And what you hear from Sundar is he believes that people are actually AI overviews will help people search more. I think what Google really want is AI overviews to increase the amount of search queries that are done now. How many other search queries result in someone clicking through to a publisher has yet to be seen. But the example, and I do believe that again you can cherry pick things. A lot of the examples and are Google's at its worst. I suspect there's a lot of queries happening right now where AI overviews is doing a good job. Right. People are happy. Again, I would love to see some data. I've heard Sundar say this in a couple of interviews that the data shows, the users show they love this. Right? But again it's interesting to hear him really highlight hey, we think this could help users search more, go deeper, become more curious because that's really the incentive. I don't think Google's incentive is to answer the query in one question. Reduce the entire search experience down to a single chat assistant who answers all questions and you don't need to search anymore. I don't think you can actually get there anyway for all of the searches that exist in the world. But I don't think Google want to get there either. So what does it mean for all of us who love the open web, who love organic traffic? Look, I think this is going to be a really disruptive time for all of us. The one guarantee I'm going to make here on the show, and I think I've got some things right about this to date, is we are only going to see AI get much more deeply woven into search, not less. Regardless of how many funny memes appear in X, regardless of how many bad PR articles and bash Google overviews woeful results. I don't believe they're going to roll it back completely. Maybe they iterate it, maybe they reduce the amount of search queries it actually exists on for a certain period of time. But I only see One direction for this, and that is Ford, Google believe this is a moment in time where they can either integrate AI into their search and win that battle or they get incredibly heavily disrupted by these other players like OpenAI, Entropic, Microsoft. The most immediate impact for all of us is going to be informational content. From all of the people I have talked to, they are starting to lose traffic on informational queries, which is that you create content to answer a question that's some form of educational content and maybe it doesn't have any real unique characteristics like data, opinion, straight up facts that traffic is going to diminish really quickly. Now I saw a really great template over the weekend from Alida who's been on the podcast and she created this and this was actually something that we were. I was talking to some folks in HubSpot and about last week and then saw this from Aliada and thought it was a really good way to go, which is this template to start to categorize your content into informational. And I do think an informational. One of the things that we're going to look at in HubSpot is really segmenting that informational into other nuanced groups, right? Because they're straight up just answering a question. Then there's educational plus perspective plus unique data plus opinion. There's like different flavors of educational content. Now in the template that I am showing what Alita has done is like she's given a template to show how you could segment all of your content into different categories which are primarily informational, navigational and branded local. Local, which is like hairdressers near me and things like that commercial, which is like people trying to buy things and then transactional. Right. And what the reason you would do that is because you would want to look at how much of your traffic is in that informational bucket, because that is the category at risk. And again, I think if you can start to look at the more nuances in there and have subtopics or subcategories for educational, you can get much more accurate about how much of your traffic is probably going to be at risk. This is something that we are going to do in HubSpot because we actually want to see how much of our traffic exists in this kind of AI overviews world and start to look to see how that performs over time to see what categories of content AI overviews is deeply going to disrupt. So that's kind of my takeaway for you is like if you actually have a bunch of traffic for coming from organic, I would start to categorize it out into these different buckets. Again, if you're on YouTube, you can see the template. I'm sure we're going to share it in the show notes. And it's Alida Solis, who's an incredible SEO. She's been on the show before. And you can start to break apart your content and your traffic in this way. And that that educational content bucket, regardless of how you break that out, is going to be up for cannibalization. I don't think Google are going to roll us back. So that is the update on AI overviews. I don't believe it has been an unmitigated disaster for Google. I believe this is just the nature of trying to integrate AI into your products right now. And Google having a ton of mindshare, a ton of visibility, and they are in a very precarious position. That's it for this episode. Join us again on future episodes. I hope you have a great week.

This data is wrong every freaking time.

Have you heard of HubSpot? HubSpot is a CRM platform where everything is fully integrated. Whoa.

I can see the client's whole history. Calls, support tickets, emails. And here's a task from three days ago.

I totally missed HubSpot grow better.