How to Say No Without Guilt

In this episode of The Diary of a CEO, host Steven Bartlett interviews Jefferson Fisher about the psychology and practical tactics of setting boundaries and saying no without feeling guilty. They explore how people‑pleasing leads to mental clutter when invitations are postponed with vague excuses, and why fabricating reasons erodes authenticity. Fisher proposes a simple formula: start with a clear “no,” follow with gratitude and a kind remark, and avoid unnecessary explanations. They discuss real versus imagined excuses, the emotional toll of indecision, and how to communicate low priority without over‑justifying. The conversation also shifts to verbal fillers, noting how filler words can distract listeners and diminish perceived impact, especially in recorded or public‑speaking contexts. Throughout, both speakers share personal anecdotes, illustrate the ripple effect of honest communication, and emphasize that saying no is a form of self‑respect rather than a personal slight.
Chapters
The last section of rule number 2 in your book is about boundaries. Yeah. What do you mean by boundaries in this context? And also, could you explain to me this, this image here? Do you know that image? I'll put it on the screen for anybody to see it.
Yeah, yeah. So this is all about how to say no, especially let's put in a context of somebody invited you to anything. Let's say somebody invited you to grab a cup of coffee.
Yeah.
All right, let's put it in terms of— that's what this, this graph is. So the graph says when you have the chance to say no or yes to something and you decide not to say no, instead you go, oh, maybe, and then it just lives in your head for the next, you know, week until they ask again. You're like, I just put it off. But if you say no when you need to say no, then it doesn't live in your head rent-free. So let's, let's put it in terms of, let's say you and I, uh, work at the same place.
Yeah.
And I ask you for, hey, Stephen, let's go grab a cup of coffee at this new place that I saw. And you don't. You're like, look, you're thinking in your head, you're like, look, Jefferson, you're nice, you know, but you're not— we're not friend friends. We're just work guys. You know, I like you at work, but you're not that cool. Like, let's— I don't want to go have coffee with you. That's what you're thinking in your head. And besides, you're so busy, you got too many things to do. And you need to say no. And instead you go, oh man, you know, maybe, maybe it just depends. I'll let you know. I'm like, okay. And then I'm waiting the whole day to hear from you. And now in your head you're like, how do I tell him I don't want to do this? How do I tell him? And now you're just thinking about it and you're just wasting your emotional energy. And it just— instead of saying no when you needed to say no.
But what if I come up with an excuse? I say, I just checked, I've got a meeting. Jefferson, so I can't come for the coffee.
Well, if that's real.
But if it's not real?
Yeah, well, that's a problem.
Why is it a problem?
Oh, because one, there's always a chance that they could find out. Two, it's just not genuine. It's not sincere. You owe it to yourself to be a truth teller.
But we don't want to offend somebody.
Yeah, that's the people pleasing, right?
Yeah. So like someone says to me, hey Steve, I know you're in Austin. I mean, this actually happens. Hey Steve, I know you're in Austin. Would be great to get a coffee with you while you're in town on Monday. And I look at my calendar and I go, I'm free on Monday, but I just don't wanna go. So what do I say to them? I just go, listen, I—
Yeah, exactly.
Yeah, yeah, yeah, yeah.
Actually, no, no, I don't. No thanks, pass, hard pass. So here's where the mistake happens, all right, is when it's that feeling of I don't wanna disappoint them. I don't want them to not like me. That's what we want. That's what really it is. I don't want them to not like me. I still want them to like me. The mistake happens when you go, oh man, that sounds so great and wonderful. I'd love to, but I can't. I have a full day of meetings. And that's all you leave it at right there. Okay. Problem with that, aside from it just being not true, whatever. The put-it is, hey man, I'd love to, but— all right, the word but has a magic of erasing everything that happened before. If I were like, look, I love you, but— and I have to say something else. Oh, I think it was great. Yeah. I mean, I think being on your podcast was wonderful, but like, it just takes away— it feels less than what you said. So instead, starting with the— like, we want to start with the gratitude. Oh, thank you so much. That sounds wonderful. I can't. That's what we wanted is like a soft landing. Reverse it. Start with the no, then end with the gratitude. So it would be if somebody said, hey, can you come to dinner, whatever, on, I know you're in Austin. I can't. Thank you so much for inviting me. I'd love to meet you or see you the next time around. And if you can, add some kindness. I've heard that place is awesome. Hope it's a great time. Like, same thing if you need to tell me no to having coffee, I can't, or I have bad news, I can't. Let me know how it goes. Like, I'm sure it's gonna be great. I've heard wonderful things about it. Let me know how it is. Like, it's much easier if you tell them that you can't upfront. Have you ever had somebody tell you no to an invitation and they're like, in a text, they're going, oh my gosh, thank you so much. I've just been so busy and, you know, I've had to go feed my cat and it's just been so stressful lately. I might be able to go. I'll let you know if I can. And if anything changes, I'll definitely let you know. Like, you're like, if you don't want to go, just tell me no. And it's that kind of feeling.
We don't know the word. We don't have the words, though. This is the problem.
Yeah.
I was reflecting over New Year's. I was at a restaurant and I was reading this book about— I think it was called The Courage to Be Disliked. And it was just developing my thinking on just trying to be a bit more sort of radically— can give more radical candor to situations. And we were sat in this, like, bar or whatever, me and my friend. And, um, the owner of the bar had come over and he's been really nice and he likes the podcast and stuff, and we're talking. And I was trying to practice this art of just being honest with, like, in those sort of people-pleasing moments. And my friend says to him, oh, we've got to, um, we've got something to get to, blah blah. And we hadn't got anywhere to go, of course. Yeah. So I remember interjecting and going, we want to leave in 10 minutes, so 'If they can get here before then, then I'm more than happy to sign the books and stuff, whatever they want me to do.' And it was so interesting how much better I felt about myself when I was just honest with this guy. I was like, 'No, I don't have anywhere to be. I just want to go.' Exactly.
Yeah.
But it's not easy on a day-to-day basis, because even when I did land here in Austin, there were people that texted me saying, 'You're in Austin. Let's go for a coffee.' And in reality, here's what my brain's saying. My brain is saying, 'I need to spend time with my partner. When I'm not recording this podcast, I need to do my—' bloody, like, my to-do list and my Slack and my emails. And then I need to go to the gym as well. So it's not that I don't want to go for coffee with you. It's that, like, I have other priorities that are pressing. So how would I communicate to them in that moment that they're just, like, not a high priority? Like, it's not a high priority for me to go for a coffee. Yeah.
So what you want to do is, like we said, instead of beginning with, oh, I'd love to, that sounds terrific, you know, but I've had, you know, I got so much to do and I can't do this. They're like, oh, okay. Instead of that, just flip it and say, I can't make it this go around. Thank you so much for inviting me. I look forward to hopefully seeing you soon or the next time. Or if you can't go to wherever they're wanting to go, you can say, got bad news. I'm not going to be able to make that happen. Thank you so much for thinking of me. Or I appreciate you inviting me. I hope it's a wonderful time.
And you don't even need to give them a reason. No reason.
No, no, no. That's where you make a mistake. When you give them a reason, you start to have to feel like you have to justify because you start— you're very subjective. We're all subjective in our head of, ah, that's not a good enough excuse. I have to give something that is worthy enough of the excuse that I'm not going to be able to make it. And then that's where the anxiety kicks in and that's where we just push it off.
What if the excuse is real? So I've been invited to this dinner in London. I actually can't go, but it's a very— someone that I care about a lot. I can't go because of my calendar. And I haven't responded to the email yet. Yeah. I think in part 'cause I'm trying to figure out how to let them—
You see how it's taking up your energy?
It is taking up my fucking energy, yes. So, in that situation, I should say— I was gonna say "terribly sorry." Yeah, no, no, no.
OK.
"I can't make it on this occasion because I'm in the Middle East." Um... But thank you for the invitation. Yes. And if this ever pops up again, please do let me know. I'd love to come. Yeah, because all of that is true.
Yeah, there it is. Okay, I think that's perfect. There's nothing wrong with saying because I'm in the Middle East. It's when it's more like that bar scenario and you don't have anywhere else to go, you just saying it, don't want to. Okay.
And don't say terribly sorry.
No, no, no, no. Yeah, because there's nothing to apologize for. You haven't done anything wrong. Save your apologies for when you said something you should not have done, you've made an actual mistake.
And I'm not terribly fucking sorry. Of course you're not.
Terribly sorry at You're very thankful. You're very thankful about it. You know, you're not sorry at all. So instead of the so terribly sorry or unfortunately, it's I can't make it work. You see how that's, when I say I can't make it work, I'm signaling to you that I have other things going on and I cannot fit it in. It's not that you're not a priority, I just can't make it work. And so anytime you get that out front and then follow it up with the gratitude, I appreciate you, Thank you. So kind of you for thinking of me. And then add on a little bit like just a sprig of kindness. I like it. I know it's going to be a wonderful time. Perfect. That's— I mean, that's, that's all you need right there. Instead of going, I need to respond to that email. And then you're like 2 weeks later, like, I got to respond to that email and you just get yourself worked up.
What is this image that I have in my hand for people that can't see this conversation right now? Yeah.
So it looks like it's verbal fillers.
So verbal fillers. One thing we've come to learn from doing this podcast is that when a guest on the show uses a lot of verbal fillers, people get irritated in the comments section. It's so interesting, but it's really taught me a lot about how to speak better. Because if someone comes on The Diary of a CEO and the guest is constantly saying like, like, um, like, like, like, like, um, like, like, it's like the top 3 comments. I just did it then. Yeah, it's the top 3 comments on the, on the video, right? Should I be getting rid of these filler words? They're like, like, um, uh, if I want to be a more effective, impactful communicator? Because the great communicators that I see on stages and stuff, they don't use like, um, right?
If you're on a stage, if you're giving a presentation, don't use them. Yeah, that's, that's a real easy rule. In casual conversation, who cares? It's the only thing that makes this different is because it gets posted and people are going to pay attention to every single word that you say and they're going to want to put that up. Everybody has some kind of verbal fillers. If they don't, they've been trained on it. That can almost be guaranteed. They've had some kind of media training if they don't use any verbal fillers, because verbal fillers are very common. But if you want to be an impactful speaker at a presentation, you don't want to have the, um, the, ah, the like, um, okay, add okay right after the sentence. Like, well, I'll see different— it's a habit for very new attorneys to begin to ask a question of a witness and say okay right afterwards. They'll ask, so do you see the red car? Just give an answer. Yes. Okay. And then they'll write it and then they'll ask another question. And but the record— so everything's transcribed by a court reporter— has okay, okay, okay throughout it. And so what a lot of senior partners will give the transcript to the junior partner and say, just review it, see how— what your verbal fillers are and all of that stuff. So it's a great way to train, but Yeah, verbal fillers. You don't want to make a habit of them. They happen. There's nothing wrong with saying like or saying, um, nothing on its face wrong with it. It just has a way of cluttering up your sentences. If I was going to say, yeah, so, um, so anyway, I mean, and you can totally tell me if I'm way off base here, but like, so essentially I was thinking, and it's, it literally does not matter. Like you see how we kind of like dip in and out and we just like get to the point. You don't want to do that on a— when you need to be very effective in your communication.
What is the most important thing we should have talked about that we haven't talked about as it relates to the work that you do, the questions that people ask you, and the value that you have to give to my audience? What's the most important thing?
That what they say, what people choose to say, has a ripple effect that will reach far more than they ever thought it possibly could. It'll affect people that they don't even know exist. How they talk to people. What you say today affects how people think of you. It affects where you go in your career. It affects how your children will talk to their children. It affects how other people will talk to their kids, and without you even knowing it. Another, I mean, example of that is the playground. Like we talked about, everybody has a memory of somebody's words It was probably very simple. The most simple words often are the most powerful. The small ones. They call eyeglasses big, huge glasses. And I remember being called four eyes. I mean, crushed. I was crushed being called four eyes. If you call me that today, I'd be like, so? But at 8 years old, I— something— oh, I have four eyes. Something is wrong with me. I remember that being something that was very important. Words will last for a very, very long time. And the people that are in your life will say something to you and you will remember it forever. People in high school, you think, oh, everybody forgets about high school. No, they don't. You go to law school, everybody forgets. No, they don't. They remember exactly what you said and who you were. And even if it's the person across the register or the person taking your order at the cafe, How you talk to them affects how they talk to their loved ones when they go home. If you've made their day difficult, they're gonna make sure that they reflect that in some sense, or it's going to put them in a bad mood. So what you say truly has the power to change everything. So it's the question and challenge of what will you choose to say with yours? If you enjoyed that clip and wanna see the rest of the conversation, you can click right here. Or if you wanna see the latest episode of the Jefferson Fisher Podcast, click right here, and I'll meet you there.










