How can an organization evaluate what their customers think about them, and why does that matter?...

Hi, and welcome back. In fact, it is a welcome back to Chris Robinson, who's come to talk to us about evaluation. Chris, welcome back. Nice to see you again.

Thank you very much. Great to be back. Thanks, Jim.

Now, you were saying that you were helping companies with their awards, but also you've got another part of your business, which is evaluation. What problem are you solving, Chris, for business owners with this other business that you've got?

Well, based The part that does awards began in 2006. And what happened is we found that to win awards, you had to do surveys and research, and you couldn't win an award just with lovely purple prose. You had to back up your claims with evidence. No better quick way to get evidence of happy customers, happy staff, happy suppliers, happy community than a survey. And a well-designed survey won awards. So we found that Nothing won awards like a survey, but we also found when especially working with L&D and customer contact teams, nothing made them do a survey which they'd been meaning to do for ages, but suddenly they had to do it because they wanted an award. So the 2 things worked really well, but because we started up just before the recession, we found that a lot of people didn't want to be seen to hire a marketing agency. And so we set up a new brand, a new legal entity called Boost Evaluation Limited, and what was great is it actually grew legs. And, you know, there's 22 people at Boost, and a chunk of them work on surveys and research, and a chunk of them write award entries. And the two things marry up. But I wanted to talk to you with my evaluation hat on today.

Great.

In particular.

Love to hear that. So tell us, how does a company go about doing evaluation so that— so it's meaningful? Because there's no point in having an evaluation that just says we're great. that doesn't manifest itself as a press release or anything. So how do you get it so that it's well balanced and also somehow giving insight into what could be better? Because that's also another function, isn't it, of evaluation?

Yeah, absolutely. And it's very interesting you phrase it like that because often, because of our heritage in awards, people would say, I want to win an award, so how do we prove that we're amazing? And you do a survey and If you cherry-pick the best news and strip out the not-so-good news and pick out all the really glowing testimonials, then you can package that up, and without distorting the truth, you just focus in on the good news. Often what we found is that people loved the fact that you put all the good news into a narrative, whether it's a case study or an award entry, and it felt great. But what they often forgot to do is what I think is more important within evaluation, and why this is relevant on a podcast that focuses on marketing and PR, is the fact fact that you need to listen, and that will improve your strategy, improve your future, and make some important decisions. And that's often forgotten because they're so busy reflecting on how awesome the past was, they forgot to inform their future. So what I was— while I was listening to your fantastic podcast and I was walking the dog, and I thought, you know, this is something I'd love to share. We as a business found the entire awards industry was rubbish at doing this, so we ran a survey, got 330 people who enter awards to tell the awards industry industry collectively what they wanted, because the whole industry was doing totally different things, guessing, firing at the hip. And, and we got a bunch of awards organizers on a call and we said, this is what people want. And, and for them it was an epiphany, and for us too, because we're in this industry. And, and it made me think, God, if we hadn't have done that, no one would have done it, and the whole industry would continue firing from the hip. And we're doing it, we're working with the recruitment company that said we want to enter an award, and I was— and they've stopped entering awards and they're focusing all their energy on a customer survey which is literally about to go out. And we're working with some training departments who are all about, we have had to reinvent training for the last year, what did we do well, what do people want to go back to, but what do they want to stick with now we've done it, do they like digital learning? And a lot of companies have had to reinvent themselves and come up with entirely new propositions. And they need to learn from the process and not just go back to the status quo and find it, but actually find that the future is probably different, but in what way? And we've seen that you need to reinvent yourself, but how? And you, and, you know, I had these I had a light bulb experience and I've watched others have it, and I wanted to share through your fantastic podcast and encourage other people to do the same thing.

Chris, thank you for the compliments. And I was listening to a podcast about the Daily Brewer podcast and newsletter, and they were talking about digital metrics and how people are looking at all the insights from Google AdWords and all the trackability. But that's only given them historical data, and that they are even suggesting get on the phone to a sample of customers and, and talk to people. So can you just share with us, Chris, evaluation— how do you do it?

Well, telephone is, is one mechanism, and sometimes people default to that. It's very difficult to schedule a phone call or get through to someone. It's quite an intrusive mechanism. It's often a bit easier to begin with a digital survey. There's loads of platforms. SurveyMonkey is the most popular one. There's plenty of others. There's more expensive ones like Qualtrics. There's some various small niche providers as well, but SurveyMonkey is a good starting point. And often you start with a digital survey, but you can invite people at the end to say, can we contact you by phone? And so it's less of an intrusion. It's also— it's as a mechanism, it is very effective, but I would always encourage you to put in an incentive. So a big part of the how, to answer your question there, is to get as many responses as possible. So incentivising a survey will get you a higher response rate, but it also gets you a balanced response because they're responding because they want the incentive. Whereas if you don't put in an incentive, you end up with what we call the Marmite effect, where people respond because they have a strong view. Either they love you or they hate you, and that's more likely to make them want to tell you their view. So you get this imbalanced response. So digital surveys, yes, forums, panel conversations are great as well. Zoom meetings are brilliant for that, individual telephone calls. But in terms of how, always bear in mind you've got 2 aspects. You've got qualitative, which is the stories, the anecdotes, adding colour to your research, and you have a quantitative, which is where the survey— you've got to have numbers. So a bit of both, a bit of numbers and a bit of colour added to that through qual and quant. There's also 2 other bits of jargon I'd like to share with you. One is objective measurements and the other is subjective measurement. Now subjective measurement are questions that you can only measure through a conversation, through a survey. And so obviously, if you're going to make a phone call and a survey, you need to focus on them. If you have a discussion with your colleagues about— begin the whole research exercise in terms of your question of how, right at the front of this is a hypothesis discussion. What is it we believe to be true? What is it we don't know? What is it we know exists, but we don't know how much or how little? And you just brainstorm all the questions you want answered. hypotheses you want tested, and then you come up with a list of questions, and those are the things that you split into— this is a subjective question, it's about feeling and emotion and their sense of the future, their votes, their experience of the past. These are sense things, they're subjective, stick them in a survey or a phone call. Then you get objective data. These are things that defy interpretation, and that's what you were describing there. These are data that you can get from Google Analytics or from, you know, you can get objective, which is somewhere between the two, like Trustpilot scores, but it's a reflection of the past and it is tracking data points. So when you are coming up with your evaluation strategy, start off with a discussion about what are we trying to prove, what are we testing, what are our hypotheses, what do we not know that we need them to vote on. Come up with that list, divide it into qualitative and quantitative, and subjective and objective. So it's 4 different bits of jargon, but it is a process. What you must never do in, in the how is just dive in and just write a survey and chuck it out there, because you'll come back to it and think, God, I wish I asked a different question, or I should have phrased it differently, or it shouldn't have been an agree to disagree like that question, it should have been—

Right.

A vote or a scale. So that's the process, you know.

Okay, I love the idea that you have to have a hypothesis to kind of prove or not prove first rather than just generally going out. What about sort of the duration? Is there kind of some sort of magic number of either questions or duration?

That's a very good point, and, and yes, dropout is a major problem. Yeah, Bain& Co. published a book called The Ultimate Question, and their philosophy is that it should be one question And that question is the most popular question and everyone's seen it a million times. It's on a scale of 0 to 10, would you recommend us to your friends? And so that's called the Net Promoter Score. It's the most popular question out there and it's one question and the whole philosophy is you just ask one question. However, I think that's overdoing it a bit. If we've done a facilities management, did a massive exercise, huge, huge survey. It was— they were doing loads of thought leadership off the back of it, so it was getting them to say whether they've done it or haven't done it. If they haven't done it, then they got questions about what they thought were the barriers and the ROI. If they have done it, they reflected on what worked and what didn't work. So it's a massive, complicated, multi-path survey. And yes, it was— we would time it and we'd say that's a 10-minute exercise on average. So make sure you set expectations. This is a 10-minute survey. But in return, you'll be in with a chance of winning an iPad. So you've got a bit of a what's in it for me. People will give you a lot more time if there's a possible iPad, or instead of a small chance of a big prize, you have a high chance of a small prize. So you can have— we've got 10 £30 Amazon vouchers, or even smaller prizes. Everyone who answers gets a free cup of coffee through Uber Eats.

The other aspect or strategy seems to be to have multiple surveys, a bit like a funnel. So have you experience in having maybe sort of a short survey with a, would you be willing to answer more questions? What about that, Chris, this sort of ongoing survey where you're funneling people into becoming almost sort of a shortlist focus group? Is that a strategy that you've experienced with as well?

Yeah, there's actually 2 paths with surveying. And companies often do both. If you're in business-to-business, which I think a lot of your listeners are, then you tend to fall into the once a year or once every half year staff surveying exercise where you survey all your customers at one instance in time. And that is a different approach to the other strategy, which is the continuous customer monitoring. If you work in a call center, and you've got one of these massive call centres in Dublin, for example, you want to track daily by call centre agent, by department, the Net Promoter Score on a call-by-call basis. So you do those classic text messages that you get, and that's— and there you would do your 1 or 2 instant questions, the vital questions, and then you'd say, can I ask you a few more? And you'd get like a 90% drop-off rate, but you've captured the critical 10%. And those are 2 approaches and they both work, but it goes back to the what are we trying to prove question at the start.

You've also raised an issue there of the sample size. Now I read somewhere that the sample sizes as small as 30 respondents can give you an indication of what the overall universe is going to think. And one notices, like for example, some of the American election surveys were kind of 100 people. Is there some experience of what creates enough of a sample size to make it valid?

There are mathematical models you can use. There's one site you put in the variables and it tells you the degree of accuracy off the back of it and/or the required sample size to hit your target degree of accuracy. And now the Very often in B2B, you have a small universe, you have 50-odd people. So how do you make that more accurate? The trick that we recommend is having a 10-point scale, because we find that statistics— whatever surveying technique you use, whether you're surveying 10,000 people or 10 people, you'll find that you get a wobble of the statistic. You keep refreshing the survey and you're monitoring it and it wobbles and wobbles and wobbles, and the wobble becomes smaller and smaller and smaller over time. until it becomes rock steady, and no matter how many more responses you get, it's really not going to change much. If you have a binary question, would you recommend us to a friend, yes or no, the wobble lasts a lot longer because one person can have a big influence on it. If you have a 10-point scale, the wobble settles down a lot faster. So if you're doing a smaller group using a 10-point scale like the Net Promoter Score, but using it throughout, the maths suggests you get it stabilizes much quicker. In terms of the number of people, it depends on your deep dive into the data. You might be that you, you are looking into splitting the data into new customers, long-term customers, and sort of slice and dice the data like that, in which case you've got to look at what is the smallest group. And I would suggest that a statistic isn't sound unless you get at least 10 responses. But if I was doing a survey in PR principles, a magazine or a newspaper wouldn't be interested in it if you wanted to publish it unless there was 500 responses. In terms of your customer survey, if you get an 80% response rate, it doesn't matter how big your customer survey is, that is representative. If you've only got 50 customers and 40 of them took part in a survey, that's the truth. So it— I dodged your question a little bit there. No, that's fine. But there are techniques to make it more accurate, and equally, it depends on the context.

Yeah. What about the survey with your suppliers, for example, and your partners, and also your staff? Because in Speak PR, we talk about 3 different audience groups, Chris. We talk about your external customers, but also your staff and your suppliers. Your partners can make a big difference. What's your experience of sort of evaluation there? Yeah, interesting.

Employee surveys. There are a lot of debates. One is about anonymity. We're dealing with a Swiss telecoms company, and they're quite upfront and say, right, it's not an anonymous survey, we know exactly who you are, and that's just our culture, and that's just how it is. Whereas a lot of organisations, particularly organisations where there might be a union involved, anonymity is essential. Also, if, like, we're doing a compliance-related survey, you know, asking people about their behaviours to do with compliance— we're doing that with one client— you really don't want to give the person the impression that it'll get back to their boss that they've admitted that they're likely to discuss the company at a dinner party. So you're asking sensitive questions. Anonymity is a big thing there, and staff surveys, you want to give anonymity. With customer surveys, you want to know who says what, and you don't need to worry about anonymity that much. You don't have to say— Don't worry, it's completely anonymous, say what you like. So they're different in that respect. And with customers, I would also encourage people to have a box saying, if we can quote you, tick this box here, because they might give you a fantastic testimonial, which you desperately want on your homepage. And rather than go back to them and say, can I have a sign-off? If they just tick a box saying, can we quote you? You can actually harvest loads of snappy, spontaneous quotes And I mean, you're a PR professional, you know how hard it is to get a testimonial out of people. And then when you do, it feels like a PR testimonial.

Yeah.

Whereas sometimes your dream snappy, the sort of funky soundbite you might get on a film poster, you're more likely to get out of a survey where people are actually being really brief. It's, oh, wonderful company, love working with you. Oh yeah, I don't mind you quoting me. And so it's very different, but staff surveys, I would encourage you be anonymous, truly anonymous. That's why, you know, this is partly why we're kept in business, because we can say we're using a third party. Your responses will not be shared with anyone within the business. But yeah, anonymity is a debate. It's worth having. Make decisions, because if you do anonymity, you lose the ability to send out reminders accurately and also to attribute problems. And there is a grey area. Sometimes you want to know which department they're from, But you just don't want to know their name so that you can go back to the manager and say, you got a bit of an issue, mate.

Chris, what sort of costs are we looking at? Can you give us some idea, even just what does it cost for Mailchimp?

Well, SurveyMonkey, there are free licenses so that you can just sign up. It'll be peppered with adverts and very limited in the number of people you can send it to. But equally, there's a whole grading scale of how much you want to pay. So if you're happy to pay £200-£300, you can get a a decent license and lose the branding and have lots of questions. You do— it's an American system, so you do have to adhere to American CAN-SPAM laws, and you can't use the mailing system to spam thousands of people. But SurveyMonkey is a great option, very, very cheap. There's loads of equivalents out there. And yeah, but in terms of hiring a consultancy to do it, you know, we just charge consultants' day rates. And so, you know, Sometimes you can knock out a survey for £500, but then on the other side of the spectrum is often a 5-figure fee for a 2-year-long evaluation strategy. So it is a bit of a piece of string.

Great. Well, it sounds like the measure is whatever you have as your original hypothesis that sort of creates the scale of the survey. Chris, if people want to find out about you, where can they find you?

Our website is boost-evaluation.co.uk. And his services are listed there.

Thank you so much, Chris Robinson, for joining me today on the Speak PR Podcast.

Lovely talking to you, Jim. Thanks for having me back.

Thanks for listening to this episode. And so we wish you the best of health, a sustainable business or a profitable one. And if you're in business, just listen and do some evaluation of your customers. But I also encourage of your team and your partners so that you can get some guidance into what they need from you going forward as we get out of lockdown. Thanks so much for listening to this episode of Speak PR.