How To Do a Successful AI Search Audience and Prompt Research

Welcome to the AI Search Roadmap series. Today about how to do AI search audience research. It seems something very trivial, but it's actually no, because of all of the implications. And to talk about this, I have invited SEO specialists with a lot of experience who I know that are working in AI search optimization as well, to share with me some of the core key most critical, craziest dos and don'ts, the top errors that they are seeing out there and how to tackle this process in a good, reasonable, cost-effective way, because we know how prompt tracking can easily make us overspend as well. Before starting, I want to thank our sponsor, SimilarWeb. Thank you very much for your sponsorship. SimilarWeb is one of those tools that I use in my day-to-day for not only prompt prompt tracking, but also a lot of analysis related with AI search as well. So thank you for the sponsorship. And to start with today's edition, I want to welcome Miriam Jasher, Andy Chadwick, and Dan Taylor. I would like to hear from you, which are those top errors, mistakes that you see people doing when developing their prompt research? See that many people are just extrapolating what they used to do with keyword research and many people just importing the prompts that are suggested by default by their AI trackers as well. We'd love to hear more and your recommendations of what to do instead.

I think just jumping off your point there, Aleyda, that's probably a good place to start. And it's the mindset shift of people approaching prompt tracking in the same way we have rank tracking historically. And I've worked with clients and seen pitches coming through where people are tracking 10,000, 12,000 keywords at the moment because they have a lot of variations. want to track the same by prompts. And it's not actually understanding that it's a different beast altogether. And also the level of volatility that happens because it's not like a linear SERP. And it almost feels like if you're trying to do that, you're going to boil the ocean. And a lot of these, as you said, prompt tools are very expensive as they are. We're just adding more and more cost to a stack and, in my opinion, getting very little ROI out of it if we do try to take this boil the ocean approach with Just to build on from that, especially on the costing side of ones, I wrote about it, I think about 2 weeks ago.

I did a test where these LLMs have context, right? All of them do. ChatGPT, Perplexity, they have context about who you are. And so if I was to search, which I did, I do a lot of CrossFit in my spare time. So I searched, the literal prompt was, can you recommend me a running shoe? I'm doing the half marathon in September. And it gave me 3 options. And out of just a matter of interest, I asked one of my friends who does rock climbing to ask exactly the same question. So not giving any context, just, I am, uh, I'm doing a half marathon in September, can you recommend me 3 running trainers? And he got 3 different results. One of them was the same, but the, the top 2 recommendations were different. And that's because of the context between both of us. I asked ChatGPT why, to give me a reason why it told me a different result to my friend. And it— I can't remember the exact reason, something along the lines of, as a cross trainer, a CrossFit person, there'll be a lot more tension on my calves and this. So it recommended these 2. As my rock climbing friend, he has a lot more tension on his heel and his Achilles, so it recommended these 2. So there was a real reason why it recommended the 2. And so the problem with people tracking prompts and the way they are, you're hitting an API. Most people are doing it by hitting an API, and it's just— it's got a blank canvas. It doesn't know who you are. And so there was a study done, and actually the article I wrote cites that study. And if you just use a base prompt with no context, the results change either a third direction over 25,000 runs. It was, it was just wildly different, up to 60% difference between each run over 25,000 prompts. That dropped significantly, and I can't remember the exact stats, so don't quote me on this, but it dropped to maybe like 20% when you gave it a context, a person. So if you append your prompts with the actual audience you're trying to target, I am a CrossFitter, recommend me a trainer, and keep running that through, the results seem to come the same more often. I think we should have always, as marketers, been interested in who the audience actually is, not trying to appeal to everyone. And LLMs have just made that— now they're actually showing it and making that bit more important. When you're prompt tracking, you really need to add the context of the person who it is. So you should append each prompt and you'll get different ones. Maybe you've got 5 different ICPs. You need to append that to the prompt. That can quickly become very expensive because then you're not just targeting one prompt, you're targeting the same prompt with a different ICP attached to each one. If you get 2 different ICPs and it gives you 2 different results and you're visible one for not the other, there's a big gap in your site targeting a specific audience. So that is interesting anyway. But if you can only target one audience, just do one audience. And that's something that needs to be appended to those prompts.

Segment each one of your audiences for each product line, service line. And yes, I understand that this can become a little bit expensive quite easily if done like that. But what is key here is the level of prioritization you have first. And then on the other hand, this should be prompts that should be really representative of your actual customer journey and the real constraints and characteristics that they search with. Because one of the main issues is I think that people try to pull the more data, the better, the more keywords, the better, and not necessarily, right?

I have many, many opinions about this because I have been dealing with nightmares. So please hang on for the top hits of errors you don't want to make because I made them for you. Number 1, a keyword is not a prompt, please and thank you. The way we used to search as humans was conditioned by the technology, by library, by Google, and now the scaffolding is falling off and we can actually talk like humans to this thing. No, a keyword is not a prompt. First point. Second point, to personas, unless you want to be testing if the model is good with Trends. Don't do that. Inject some context. Let me give you an example. If I ask these two gentlemen what a naked shoe trend is, they're not gonna give me the answer that the haute couture Paris maisons are expecting. Okay, so some of these models are really good to catch on to these implicit trends. Some of them are disastrous, bringing some trends from the '90s or like orthopedic stuff that doesn't belong there. So please inject. The context of the person when you're trying to do this. Otherwise, you're going to get a lot of noise. Okay? What happens if I'm one person analyzing a bunch of prompts, but that are in Japan, in China, in Germany, and I only speak English? You are going to have to make some choices. This means that your data will be biased. Okay? It will be because it is what it is. The prompt language impacts the sources that are going to show up. But But, and this is important, it has to fit what you're willing to do with this. So if you think your prompt library or taxonomy is the endgame and you're proud of yourself because that's a beautiful thing, no, that's the first step. Okay. I have a few more. If you're waiting one week and then looking into the results, you're doing it wrong. Please wait a little bit. Be patient. At least 30 days. For goodness sake, we're professionals here. Okay. Tell people to chill and to wait. It's a living document, so don't, don't let it die because some of these prompts, they're going to change, which brings me to the last thing I want to say here. If you are going to track some trends, like I said, you know, some things that are important, validate the trend, please, because otherwise it doesn't necessarily deserve to be here if it doesn't make sense. If you had outdated historical data, it's not going to be representative. And last but not least, when it comes to mistakes, that's the mistake number one everyone makes. What you're doing is an ideal scenario in a sanitized lab space. You are looking at the best-case scenario you could have, not reality. So please keep this in mind because this is giving you a direction. It's not giving you the full reality.

I love that last part regarding how what you see in any case should be directionally helpful. And indicative of gaps and opportunities rather than making you to obsess of whatever instance of these prompts where you are not showcased, right? There's a level of inconsistency that will tend to happen. That is why what you are looking to identify are patterns at a topical level rather than obsessing over individual behavior of prompts as well. We need to evangelize. I love how you have shared. The actual most common errors, mistakes, issues that we're all facing. I would love to hear what are the actual steps and criteria that you're using right now to create a good prompt library, to understand, to have a good directional reference of the behavior of your users in AI platforms, and then to be able to measure how your brand is performing for it. And based on that, to identify opportunities, to identify gaps, because we, we shouldn't forget why this is important in the first place.

One of the things we're doing a lot of is just trying to, again, help clients better understand volume and quality and actually getting ROI. Because again, I think for the last 10 to 15 years, we've trained everybody on more is better, more keywords, more traffic, more everything is better. I mean, obviously more traffic is better. Let's not get it wrong. Doing things like downsampling. And have an evaluation dataset. So we don't need to do 1,000 prompts or 1,000 variations to run test cases. We can just do really optimized smoke tests, 20 to 30 prompts across libraries, daily iterations, and then use them to put together a VUBI end kind of release candidate set of prompts. We actually then do track across ICPs. Even down to using smaller models. Everyone who I work with, a lot of companies I work with in the US are kind of drawn into always upgrading to the most expensive models for tracking. We can use the cheaper models to run run little test cases as well and judges just to see if there's any legwork in it. But then also I'm a big fan of putting together rubrics of value at the moment to actually understand if smaller models are producing certain key trends across a wider set, it's cheaper to track. And then we take those candidates and those prompts and actually run them on the larger models, which are more likely to be used by consumers. And then alongside that as well, actually understanding that not all users and audiences will be using the same kind of LLMs. I do a lot of work in B2B pharmaceuticals. Yeah. And most, I mean, I do a lot of B2B upscale pharma in the US, the kind of stuff where not direct chemicals and things, but if you want to buy lab equipment, for example, you want to buy a million face masks or something. A lot of people are on lockdown company computers, locked down to using Microsoft Edge, God bless them, and using Bing and Copilot for everything. So Copilot is more important to them than anything else, but all the tools, but mainly push on ChatGPT, Gemini, et cetera. So it's actually refining the stack to work for them as well.

That's a really good point. I read another study, I think it was by Ran Fishkin, the drift between the different platforms. So I pulled up the stats here. So Perplexity, the monthly citation drift is 40.5%. And what I mean by that is it's the steadiest. It gives you the same for a given set of prompts. 60% of the time it gives you the same answers. 40% of the time it changes. Copilot, 53.4%. ChatGPT, 54%. Google AI Overview, 60%. So that they all drift wildly at Google AI overviews being the most erratic. And then what I did, and I pulled this last week, is to look at where they're pulling from. And Sigamthan wrote something on this as well, my co-founder. So Perplexity normally pulls from live web search. Traditional, I mean, I know we hear this a lot, but the tactics change for each one. So for Perplexity, it's closest to classic SEO and digital PR. It's the underlying rank, which it's pulling through. ChatGPT is more around consensus. So it's more of those listicle type things that people are telling you to go on Wikipedia, that kind of thing. Google AI Overviews has a 54% overlap with classic organic rankings. So that's back to traditional SEO. And I've got one for this. Each platform actually has slightly different optimized, call it optimization technique, but optimization type thing that you go for. And when we, when you look in your whatever platform you use, PKI or whatever it might be. and you blend them, that's not really the right thing to be doing. You should be reporting on each platform. And as you just said, some have a different audience, like your, your developers are using this. So this is, and this is where this pulls from. So this is what you should be doing. So that's just wanted to build on that point there.

So you're going to open Also Asked or an equivalent. Okay. You're going to get in touch and make friends with the social media team. If there's no social media team, you're going to do the research on your own. Okay. You're going to open Google Search Console. You're going to open Google Analytics or an equivalent. You're going to make best friends with the traditional marketing team, so they give you the personas. Okay. You're going to ask what the actual business objectives are. Once you have all of these sources, and the way I like to talk about it is that you have your historical sources, which is your Google Search Console, your Semrush, et cetera. That's a historical diagnostic, you know, your internal analytics. You're going to get your market intelligence by asking what the campaigns are, what the personas are, et cetera. You're going to do the latent research, which is what are people saying with Also Asked. You're going to use Google Trends, maybe you're going to look into social media listening. And everything that you're going to find is going to help you figure out what these prompts should be. Okay, the reason why I say this is that if you look only in one spot, like Google Search Console, you don't know what you don't know. And you're not a social media expert. But oh my goodness, are third-party websites really feeding these answers. So you need to become one. Okay, so you have to get confident and actually speak the language of the people you're trying to reach, right? Because if you don't have the lingo, it's not going to happen at all.

Okay.

So once you have all of this, you're starting to understand what's going on, you're able to structure your questions a lot more. So this data, it's nonlinear. This means you will be going back and forth, you will hesitate, you will have to prioritize, you will have to segment it, figure out what it is, what it does, what it's supposed to do. And there is one question at the end of this big first trial that you have. You have to ask yourself out of all the prompts that you have, if AI answers something really bad about me here in this question, is it going to cost me a sale? Yes or no? That can be at the beginning. That can be a brand thing. If it says, hey, don't really trust them. I have to show them because they are the main ones you asked me about or they are discussed in this space.

But don't—

will it cost you a sale? If it actually tells you it's not for you because there's missing information, that can cost you a sale. You pay attention. That's the one question. But beyond that, there's another sneaky question you should be asking yourself. Is there sentiment bias in my prompt? This is important for you to know. That doesn't mean it's going to be tagged in the tool. That doesn't mean anything else. But when you're going to analyze sentiment, you're going to come out and say, oh my God, we have terrible sentiment. How do I explain this to my boss? No, you asked like, is this, can I trust it? You already are negatively biasing the answer by asking a negative question. So be aware of the sentiment in there as well. Okay. This is one of the big things that I see not happening. And I'm like, and then you're surprised it's negative. You're probing for negative stuff.

Thank you very much for bringing the negative sentiment since this is assessed across most tools in the most simplistic way. So for example, If you have a certain type of brand that has a very unique selling proposition that might be perceived as something expensive, I have seen many LLMs depending on the wording, right? Like, oh, this is, this is expensive. So it's bad for you. And they flag that sentiment to be negative, but actually it's not because it aligns with the brand unique selling proposition and values. And it's just that the persona or the audience is not the correct one, but that is why you need to really provide the context, the right context. For measurement and also something very important that you highlighted. Unlike traditional search, AI platforms can be seen more beyond not only a performance channel, but also a branding channel. And that is why there's always this share of valuable monitoring prompts, I call them, besides the core prompts that these are the more stable ones used for recurrent tracking, reflecting the highest priority journeys that should remain consistent over time. I also have like usually the experimental ones to test new products, markets, constraints, audiences, emerging behavior, things like that. But also the monitoring ones that to track reputation, brand representation, risk compliance, because it is actually not only a retrieval system anymore, but a recommendation engine that affects branding. And that is why we need to track all of these additional behaviors. What we always need to think is that You should always ask why, why am I measuring this? It should always connect with an action on something that is worthy to optimize for, a goal that we should pursue. If it doesn't connect with that, then it's usually, or very likely pointless. And that is a good way to prioritize accordingly what you should actually measure and not.

For my SEOs out there, volume and ranking. Okay. These are 2 things. Your prompt volume is based off of certain things that are very interesting, but They're not the same as SEO volume. And I will let my peers talk about that all day. But the second element is that you also have to condition the people working with you. I'm going to put in the prompts and we are going to prioritize them. Okay. We don't, we can't track more. It's going to cost us more. So people will still want more. You have to tell them, no, we have a system. Here are the priorities. If you want to add more prompts. they better go through and be validated and really belong to this. So if you don't have someone that is responsible for the prompt, don't put it in. What do I mean by this? I have some sales questions. Well, this is going to go to the customer experience team or the support team. They own fixing that stuff. They own the data that comes out. I'm facilitating. I'm giving them information, but they are responsible for improving this. I can guide them, but it's their baby. So There are some prompts that belong to marketing. There are some prompts that don't belong to marketing and they have to be in there because otherwise the company is not thriving, but they should be someone's responsibility. And as SEOs, we tend to think that we can solve all the problems. No, that's not how it works.

Yeah, we cannot work in a silo, 100%. We need to align with all these many other areas and even expand further our alignment indeed. To start wrapping up, I would love to hear your last tip, but last resource a last action that you think we haven't yet covered and you would love to recommend the audience to use or to leverage or to think about when doing good prompt research or when establishing their representative prompt library.

It's not something you can probably transfer across all businesses, but in where you have this data available, I found it extremely useful to do. And that's to use Google's own analysis it gives out to the public against it for the prompt side. So if you have a business which has a lot of good Google reviews. Google already runs heavy sentiment analysis and callouts on it. So you can go to reviews and it will pull out for you when you're looking at it saying like lots of people meant— if you're looking at a hotel, for example, lots of people might mention the pool's good, but the restaurant's bad kind of thing. You're not going in blind in trying to think what people are saying or talking about you. You can harvest that from your own reviews. And Google's already kind of done heavy categorization for you. So you can pull that out and use that as a baseline to then actually start to put together separate libraries of prompts for your tracking and testing.

More of a mistake that I see a lot of people doing, because when you plug— you said at the beginning already, when you plug your brand into one of these fancy tools, they come up with a lot of the prompts for you. I think I sound like a broken record. My first one is, as multiple times we've touched on this, the persona is really important. Coming up with that persona, which has always been marketing anyway, but now LLMs are punishing you if you don't really know who your persona is. So that's a big part of it. But the other mistake I see people make if they go for those pre- written prompts that come up is the way that they're generated. They're often basically building it on a library of content you already have. So the tool is reverse engineering the piece of content or your URL and then coming up with the prompts based on the stuff you already have. So it's almost just self-fulfilling your own prophecy. And then so you'll put them all in, it'll say, oh great, you've got— as I've said, I don't like this metric. You should break it down per platform, but 60% You know, oh great, I don't need to do much. Well, the way it's come up with those prompts is because it's reverse engineered what it should rank for anyway. So you should be high. So there's a trap that I see a lot of people fall into as well.

I do have a tip. So once you have picked your prompts, please use the tagging system. Please use the segments. The reason why I say this is that you could have a really, really good sentiment overall, and then you dive into customer experience and it's a doozy. So this is really important as well. You have to think about your prompts. Think about your prompts as not only an artifact that you're going to put in there, but hey, you're collecting data out there. How are you going to analyze it? How are you going to, you know, make it actionable? So please be kind to yourself and actually think about that system. Because otherwise, you're going to end up crying tagging it by hand, or uploading another CSV thinking, okay, now my tags are in, and then you actually duplicated your prompts, and it's costing you twice as much. And this is a confession. So yeah, don't be me.

I will highly recommend everybody to start with a minimum viable list for a single, your more important audience, your more important product or service line. And once that you have that tackle and you validate that is what you want to accurately assess and understand, you can take it from there. The problem that I see is that many, many times we get quickly overwhelmed because we want to start with everything. And of course, if you are Working with an e-commerce brand that has dozens of product lines, it's very challenging. Another thing, we thankfully are getting more and more data, first-party data, rather than just relying on third parties. So a big, a big thank you to the Bing Webmaster Tools team because they have been the pioneers in this area. They are giving us more, not only prompts, but their topics and the intents behind them. Again, Please apply the recommendations that we have all shared before rather than obsessing over the wording, et cetera, because it's not for that. It's to identify patterns, to understand better trends or the trends behind the behavior. They can be used and should be used very much as an input to create your prompt libraries. The same with also Ask, speaking about Mark Williams Cook, also wonderful tool, can be used for this as well. And if you have a little bit of more But yet there are also tools like SimilarWeb as well that show the prompts that are generating you AI traffic. But of course, these are the very bottom of the funnel. They are not necessarily representative of all of the prompts that are being asked out there, which is one of the other mistakes that I see people doing, right? Oh, because these are the ones that are generating or are meant to be generating clicks. I am going to only focus on this, but we know that is not how AI platforms work, that you want to get cited. And these are not the only ones that are actually representative of the whole customer user journey. Indeed. Thank you all very much for your recommendations, for your tips, for your insights, the mistakes. Hopefully this helps SEOs, AI search specialists out there to advise their clients and to show proof that they are not the only ones going through all of these challenges. You can also follow them because there are always Sharing a lot of resources and insights about this and other AI search topics. Thank you very much for your attention today. I hope that you have found today's episode very actionable, very useful. This is one of the many more episodes to come into this new AI search optimization series. I want to thank again our sponsor, Similarweb, for today's episode and If you don't want to miss the other episode, subscribe to the channel and follow again our today's experts in order to learn more about AI search optimization in the right way. Until the next edition, bye-bye.