#26 – ChatGPT for CRO, Will AI Replace Marketers?

Fundamentally, what you want to keep in mind is you don't have to take everything that ChatGPT suggests and just apply it blindly. But it could raise interesting points because you still need to use your mind. Nothing beats the human mind, at least in my opinion. You're like, okay, this is an interesting point. Maybe I can modify it and work on it and optimize it further. My name is Kalislade, and I love to talk about conversion rate optimization. I'm Simba and I love asking questions about conversion optimization. This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization. Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal. Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we'll talk process, and we will talk tactics. Simba will be bringing all the questions. Oh, man, I bring tough questions like do a B testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions. Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts. Another week, another CRO Live Hour. It's been like a year and four months of doing the CRO Live Hour. So super excited. And it's funny, you think at some point we'll run out of things to talk about, but there's always more and more interesting questions that come on the inbo. The best side, maybe we've used Google Optimize only once in the last, I don't know, eight years. Most of the companies that we work with are larger, and even if they're small, the type of experimentation and how many experiments we launch per month, it made Google Optimize just not a real option to use. But what's fascinating is the fact that even as people who use Google Optimize start taking a look at figpy, they're surprised by some of the features that they see and I'm surprised by some of the stuff that they tell me. So, for example, figpi does very sophisticated revenue calculation to help you understand the impact of any experiment that you do on the bottom line that we knew that it was not there in Google Optimize. But even matching things between Google Optimize and Google Analytics was an issue for companies who are very serious about the testing program. And of course there are features that are in Fake Buy that's just out of the box, are not there in Google Optimize, Heat maps, session recording, on site polling, and of course there's the recommendations, which was never there in any of the other A B testing platform. So it's fascinating, it's interesting time to be living in an invest or click buy and it's fun as well. And I think today is going to be interesting because we are talking about ChatGPT. Everybody jumped on the bandwagon of ChatGPT, so we might as well talk about it and how to use it when it comes to conversion rate optimization experimentation. A B testing so how does ChatGPT impact conversion optimization? There are three different ways that ChatGPT AI can help anybody who's working conversion optimization and experimentation do a better job in improving a site conversion rate. Those three different areas are hypothesis creation. And when we talk about hypothesis creation, we're talking about more sophisticated hypothesis creation. Number two, they can help you in terms of copywriting, which is great, and delve deeper into what that means. And number three, they'll help you in terms of a B test development. Those three areas, if you know how to use ChatGPT in the correct way, can really enhance your job as somebody who is focused on conversion optimization and can deliver better results whether you are working internally or you're working with clients to try and help them improve conversion rates. Foreign. Let's jump right in and talk about those three different areas. The very first area is hypothesis creation. Now, typically, whenever you're creating a hypothesis for an AB experiment, there is a step that comes before that. That step is identifying a particular problem on the page. That's number one. Now, if you're lucky and you've done some A B testing, you also have some A B tests results that you've collected. You've ran some observations about the website visitors. You also have some best practices maybe that you apply because you've done conversion optimization for a while now. What you can do with ChatGPT is take all of these input points. Here's the problem that I'm trying to solve. Here's what I've learned about my visitors. Here are some observations that I've done based on my previous AB testing, and here is maybe an initial hypothesis that I have. And you can ask ChatGPT to give you suggestions for other hypotheses that you can run on the particular Page or additional hypothesis for AB testing. Now, when we experimented with that, it was interesting because sometimes some of the hypotheses were interesting, sometimes they were not so great. And I'm like, really? Okay, because again, it's AI, it's trying to figure its way around. So when you don't get a good hypothesis, when you're like, come on, chatgpt need to come up, by the way, with a fancier name for it. But come on, it's like, oh, this is not good. A good conversion optimizer will provide that input to ChatGPT to say, well, here's the issue that I see with the hypothesis that you've suggested. Can you modify it in this particular way and work on it and optimize it further? But ChatGPT can be extremely helpful in that way. That's number one. Number two, ChatGPT can be very powerful when it comes to copywriting. And copywriting is extremely important to any type of conversion optimization. Lots of times whenever people work on optimizing a website, they are very focused on the design. But I always say design is important, but it comes second to copywriting. We put up with ugly designs if the copy on the site and the messaging on the site is persuasive and captures our interests. And ChatGPT can help you create that copywriting. You can give it some of the headlines that you are thinking about, so it can give you different alternatives, you can give it some different sections in the landing pages that you are working on, and it can give you suggestions of other things, other ways to present that copywriting. Again, it's rare for ChatGPT to be 100% perfect, but at that point in time, your job as a conversion optimizer and your job as a copywriter becomes more of an editorial job. You can take the copy, you can fine tune it, wordsmith it. So it's not highly intensive AI written copy, but AI. And ChatGPT can actually help you in creating maybe 60, 70% of the copy that you're writing. And then you do the worst Smithing. You do that editorial job and you end up with very powerful copy. And since we're talking about copy, we also have to keep in mind that ChatGPT can help us create copy for our ads, whether on Facebook or in Google Ads or Bing or any other platform. You can supply ChatGPT with different headlines that you are thinking about and see other variations, other suggestions that the tool can give you. So that's two different ways. They all are around the copywriting portion of what's a CRO, or Conversion Optimization Specialist tool? The third way in how ChatGPT can help you improve your job as a CRO is in actually writing the code. Chat GPT has programming languages written into it. Most AB testing is done using front end code to manipulate a particular page and you can actually ask ChatGPT to write some of the code for you. Now, the code is very powerful when it comes to ChatGPT, but if you are doing something complex, don't expect ChatGPT to write everything for you from 0 to 100%. But it might help you really get about 80% of the code that you want, which is very powerful because that means it is something that's going to allow you to do your job in a much better and faster way. Again, similar to how you did. When it comes to copywriting, you're performing the editorial job. When it comes to copywriting, when it comes to writing code, you're performing the senior developer job of looking at the code, optimizing it further, saying, you know what, I can actually take this and improve it a little bit, a little bit further. Now all of this by the way, is around ChatGPT. One thing that we are doing that is very interesting away from ChatGPT is machine learning. Think about the job that a conversion optimizer does. They look at inputs and those inputs could be analytics, could be heatmaps, could be session recording, previous AB tests. They're doing some sort of analysis to come up with that initial hypothesis. Machine learning is ideal for a job like that, where you can use machine learning to look at all these different inputs and then based on that create a hypothesis. And we've done enough A B testing that we actually have models correct for things that we can apply machine learning for, come up with a final answer and then run like take that machine learning model, run it for new websites. So really the analysis can become a lot more sophisticated than your typical analysis. The beauty of this is we don't have to invent the wheel. Like, you know, reinvent the wheel or start from scratch. AWS has a very strong machine learning algorithms that are specific to E commerce. So we are working with AWS and actually saying, hey, how can we use machine learning to help us improve the job that our CROs do? This is machine learning a lot different than AI. AI is just figuring out the road on its own. Machine learning, you start with the roadmaps like, here's the map, here's point A, here's point B, here's the correct route, how I can get there. It's just different and fascinating ways of how you can work with a website to improve its conversion rates. The very first point that I mentioned, ChatGPT can help with curate the hypothesis or an experiment. But your hypothesis is as good as the inputs. And that's the reason lots of people when they start providing input that is not very clear input, that's maybe still like in a kind of like not black and white, more maybe on the gray area. They're complaining that the hypothesis that they're getting from ChatGPT is not necessarily very powerful. That's the reason I would say it's natural to have not so great Hypothesis. Don't expect ChatGPT to do your job for you. You still gotta use your brain. It's almost like having an assistant. Correct. That's gonna give you some suggestions. Sometimes you look at your assistant and you're like, oh, they're way off. And sometimes oh, this is really great idea over there. One of the partner agencies that we work with are using ChatGPT, which is extremely powerful in creating e commerce product descriptions. They are working with a client, 4,000 different products. And one of the biggest challenges that e commerce websites have is that the product descriptions are just really either taken from the manufacturer, which means that you have duplicate copy. I'm using the same product description that any of my other competitors are using or if they want to create it, it's just a humongous job. So they said, you know what, ChatGPT is really great. It's great for creating all those product descriptions. Now you don't want to just take those product descriptions and just throw them on your website. You still have to perform that editorial job to make sure that you have actually good copy on the site. So in complex analysis, when it comes to analytics, complex analysis for heat maps, again, chatgpt is a text based app. Correct. So can read actually a heat map and make a recommendation. I love that. It may be even just the initial process of describing your products, the problem that you're trying to solve for people and then telling here like now I want to create a poll asking people this type of question. What should I include in it? What are the options that could be interesting. Here are the outputs correct from the poll. 40% of my visitors are complaining about this, 30% are complaining about that. How do you recommend what you recommend? Again, it can be a sounding board for you to think about new things that you can launch on your website, new experiments that you can play with. The less sophisticated you are Eventually AI and automation is going to replace you. The more sophisticated the work you do, the longer the road ahead in front of AI to be able to replace somebody and the reality of it. Think about copywriting, copywriting, it's an absolute amazing gift. Some people have it, some people don't have it, Some right. And ChatGPT can help create some copywriting, but you can tell to some extent whether this copywriting is written by a machine or is written by a human. And lots of times, this is not to say like lots of times I read copywriting from people who specialize in copywriting and I think to myself, I'm like, oh gosh, this is just typical, what I call marketing copy that people read through and sniff through and it's just not going to convert them. Good copywriters are rare to find and I absolutely, by the way, spend a lot of time studying their work and just fascinated by how they say things in a way that I'm like, oh man, that sounds just so beautiful and it's just so captivating. I can tell you that ChatGPT probably will replace 70% of the copywriters out there, but the top notch copywriters, they're probably not worried. As a matter of fact, they think to themselves, you know what, ChatGPT and AI and all these tools are just going to help me become better copywriter. Sometimes we work with companies and I look at some of the manual processes that they have and outdated. They have four or five people doing them and I think to myself, you know, all these people could be replaced with an automation script that the developer can write probably in less than a week. That's a scary thought to those people because they don't understand that you could be replaced easily. And this reminds me of one of the architects that I've worked with with very early when I was first starting to do software development and he had the sign on his desk where he said, go away or I will replace you with a script. So you really had to think through before you ask him a question. And when you ask him a question, needs to be a smart question. The better you are at your job, the more difficult will be for automation to replace you. And as a matter of fact, automation will be a tool for you to become better at your job. We do quite a bit of content marketing. That's how we've built the company. If I'm merely using AI to create content, great, so I'll throw more content. But guess what? I'm relying on Google. Our most amazing content is not requires two pillars, two elements to it, the content piece, but then there's this distribution and the distribution is very hard. Yes, I can use AI to write some content to like not do the outreach and whatnot. But most likely you're spamming people. Not very helpful. The distribution side of things is the human relationship. Correct? Our investors distribution comes from like oh I know. So and so I've written an article about Google Optimize and one of the largest sites about online marketing. They publish one article and I'm like, I can do so much better. But their editorial, to get through their editorial process, it takes forever. So I Ping the CEO of that company because I've known him for 15, 16 years and I told him, hey, so and so I've written an article, what do you think? Can you guys publish it? And he got back to me just actually before our LinkedIn live and he's like, yeah, he's like, let me connect you with the editorial team, you know, let's chat. And at the same time he said hey, like, you know, by the way, I'm working on this landing page with a client, can you help? And I'm like, of course. Nothing beats the human relationship. AI is not going to do that for you. That human relationship. Next week I'm flying to Popcorn Austin. I'm going to be meeting a whole bunch of people. I'm so super excited by the way, to be meeting with people. I'm not sure, like you know, we're doing a dinner on Monday, this coming Monday with a whole bunch of the CROs. So it's always interesting to chat with them. So I'm going to be asking them to see. I know Brian Massey is going to be there. Pep most likely will be there a few other people. So it will be interesting to have a discussion with them on ChatGPT and see how they are using it in their agencies and in their work as well. Good agencies should be always looking ahead and asking the question, how can I improve something? How can I improve my current process? When I hire a senior CRO, One of the biggest things that I tell them we have a very well established process. I was reviewing a little earlier, we have 17 different steps to creating an A B test. 17 documented steps prior, by the way, to coding the A B test. And the reason I've documented that because sometimes when you work with a company, they don't appreciate all the work that goes into an A B test. They're like, oh, that's very quick. I'm like, no, it's not. It's actually a very demanding process and what you end up with before even we code anything a 21 page document. 20, 21 page document. Our test briefs. So there's a lot of work that goes into this. Whenever I hire a senior CRO, I tell them, listen, we have a very well established process, however, and it's good to have a process. I will take a process over just chaos any time of day. However, since you are senior, I want to see not the first month, but like second month. I want to hear from you three to five new ideas of how we can improve process, how we can do better. And I get very interesting suggestions. Even more fascinating is as we start working with different companies, some of the questions that they ask, and they're just merely asking a question really can inspire changes to the process to say, hmm, yeah, this is an interesting question. So we've modified our process, obviously. We've introduced three or four new things to the process around what type of testing we do, especially when companies stick with you for a while, what kind of testing we do, what kind of analysis, what kind of insights that we get. Now, if you're just merely starting to work with a company to try and help them improve conversion rates, the first 12 months are easy, by the way. The second 12 months are easy. But if you work with a company since 2011 or with another company since 2017, the process is a lot harder. You've tested many things, you can go back and retest. But, okay, so, yeah, that's fine. That's maybe like 20, 25% of your ideas. But how do you innovate and how do you come up with new testing ideas that really push you beyond your comfort zone? That's always fascinating. So we're looking at design patterns, we're looking at like, you know, specific templates that we use. We're looking at some kind of what I call away from just tweaking the templates into, okay, how do we actually test to improve revenue? This is always fascinating, but if you're stuck doing the same thing day in, day out, you're not going to improve. And if you don't improve, you're. You're digressing. We have one client that I've been working with for a long time. This is the fourth or fifth management team that we work with from that client. They've had turnover over the years. I mean, you work with a company for that long. I was talking with their CTO and it's fascinating. By the way, one thing that I've noticed in the last, I would say 24 months is many of our conversations are happening with the CTO, not with the CMO. About 70% of our conversations are with the CMO or VP of marketing or the founders. Depends on the size of the company. But 30% of the conversations that we're having nowadays are with the cto. By the way. Completely different conversation. And it goes back into blurring the lines nowadays between CTO and cmo. So that's a whole discussion for another time. But we're talking about the type of test and the priorities that the CTO has are different than a typical cmo. Priorities. Our goal as a company that does conversion optimization, our goal is to help the company make more money online, Improve your conversion rate, your revenue per visitor, your average over value. Well, the CTO has a different set of priorities as well because he's looking at site stability, he's looking at code architecture, and he wants the company to continue making money online. And then, by the way, this is not to say anything, but sometimes explaining a B testing to a CTO is easier than explaining it to a marketing person because there is a technical side to ab experimentation. So he's like, yeah, I love this. And then initially he came in and he's like, yeah, what's the value? After we explained it, and he probably, I'm guessing because we had the first call and the second call he had dug through and he's like, yeah, this makes perfect sense. However, I have also five concerns that we need to address and I would like to test this, this and this and how, like, you know, he's trying to understand our methods of actually plugging the data. I was telling him, like, oh, this is an amazing conversation for me coming, like, because it's like kind of tickles my background in development. I'm like, oh, God, this is nice. So let's talk about this and let's talk about that. So it's just always fascinating. We focus a lot on data. I really mean we focus a lot of data because we're constantly digging through analytics and pulling data from four or five different sources and kind of looking at them and kind of like, you know, different angles. We used to say, you know what? Clients would love that. But we've worked with companies who could not care less, by the way. They don't care about data. Just tell us what you're going to be testing. Let's move on. If you try and have the data conversation with them, they're not very interested in that. So for those Types of clients. We're like, okay, we'll do our work correct, but we don't have to present all of these. We simplify it. We present it. We have other clients who are so data focused, who are just basically, we can. We'll also spend hours with us kind of talking about data analysis and digging. And it's interesting because with some companies, they want to, like, you know, slice and dice and look at different angles to make the right decisions. With other companies that, like, just tell us or just simplify it because this is getting too complicated. So not only that the process changes, but how and what we present to different companies that we work with changes based on the company because you want to find the good match for those companies that you're working with. I was talking to one agency and he was telling me that about one of his account managers who had mastered the arts of dealing with clients. However, the actual work was not good. So he's like, you know, we know that the work is really not good, but this guy has really good relationship with the different accounts that he's on. But he, as a manager, he said, no, our work quality is much higher. We demand a lot more. So they end up letting go of this guy and he's like, we know that the quality of the work is not there. Within one week, we lost four or five clients because they all had the amazing relationship with this guy and they all wanted the relationship more than the actual quality of work. Again, different clients want different things. Some clients are very focused on the relationship. Some clients are focused on the quality of work. I always tell our team members that the secret to success when it comes to conversion optimization is both you have to deliver amazing relationship and you have to deliver amazing work. That's how people stick with you. You do one versus the other, you're going to end up losing the company. You know, you do just relationship without results, without really good work. They're like, yeah, good talking to you, I enjoy it. But no paying you too much. Or you do the other way around, which I think will get you longer with a client that you're delivering good work. But if you don't maintain the relationship, it becomes very mechanical. No, you want that personal relationship. So it's kind of a mix between the two. Actually. Earlier today I was talking with one of our lead developers and I'm like, well, this is the last point. And then this is the last point. I'm like, okay, this is the fifth last point today, but I want to ask you about this. So, yeah, Go for it. The objectives of the client might make you say, you know what, we don't need to focus on this. The specific requirements of the client and the specific industry of the client sometimes will make you say, you know what? We don't have to. So we do competitive analysis as part of our conversion rate optimization work. It's a cornerstone. We want to understand the market. We're not copying competitors, but sometimes we might steal some of their ideas and we validate them. If you work with a company that has no competitors, and people always say, really? No company has no competitors. Yeah, we worked with a couple of companies where they're really defining a completely new space where, hey, it's not like, oh, yeah, this is just like everybody else had done this. No, it's a completely different way of doing business. There's no direct competitors. There is the current status quo of how people do things, but those are not direct competitors. And you have to understand how those work. But you're not doing the deep competitive analysis that you typically do. I'll give you an example. One of the companies, I had a conversation with them and they're in air cargo. Okay. Typical space has been around for a while. You have all these airlines and, you know, you can reserve like, you know, space to shift your cargo. What they've created is a marketplace for air cargo where people just like when you have a marketplace for gas, for oil and gas, and there's like, you know, future contracts, they have the same thing for air cargo. That's a completely new space. I mean, it took like five minutes for me to even understand the concept. And I'm like, wow, this is fascinating. Okay, you've created a new markets opportunity. So there's no competitive analysis that happens, that happens in that area. You still have to understand the current status quo and what people are doing, but you're not looking at direct competitors. Analytics and what we do with analytics, we spend a lot of time on analytics. But our approach to analytics, the data that we have to pull for analytics is a lot different. Even how. And the requirements that we have around launching and maybe experiments, we have very strict requirements in terms of determining sample size that is required. What we call early stop measures, late stop measures, all that stuff that we make. If you're working with a startup that doesn't have enough data, you have to be flexible. The best example that I tell people, if I'm working on a new medicine, you're a startup. As you're rolling out the new medicine, you're starting with a much smaller sample to validate, and that's what we do with those companies. So you really have to be flexible, understanding the objectives of the company, the context that they operate, and you have to adjust your process accordingly. I enjoyed the conversation about AI and ChatGPT. We'll come back next week. Well, thank you everybody for joining. As usual, if you enjoyed this conversation, make sure that you like and subscribe. If you have any questions, drop them in the comments, reach out to me or Simba and we'll come back. Thank you everyone.