Can You Retire at age 50 with $5 Million? The Answer Isn't What You May Think

Can you retire at 50 with $5 million? One person absolutely can. Another person with the exact same $5 million may not even be close. The difference is not the $5 million. It's everything that number has to support. Welcome back to the Left Brain Investment Committee. I'm Brian Dress, Director of Research and an investment advisor here, along with Nolan Langford, our CEO and Chief Investment Officer. Nolan, welcome to the program.

Hey everybody. We're talking about our favorite topic, retirement, with a special twist, early retirement.

Well, I know you're not gonna be an early retiree, but some of our clients might wanna be. So we're gonna talk more about that today. We hear some version of this question all the time. I've got $5 million. Can I walk away? And it is tempting to give people a magic retirement number, but 2 50-year-olds can have the exact same portfolio and get completely different answers. Today, we're going to show you exactly why that is. We're also going to talk about what working another 2 or 3 years can actually buy you, and one part of early retirement that has nothing to do with the spreadsheet. Noland, why is $5 million not enough to answer the question?

Because it depends.

Our favorite answer.

It depends. You know, Brian, a $5 million portfolio supporting $150,000 spending, Or 3%, much different than a $5 million portfolio, uh, supporting $300,000 in spend. So it depends. Perfect.

So we're gonna set this up the way we're gonna get started. Same $5 million, 2 different households. So Household A, age 50, $5 million invested. They need $150,000 a year from their portfolio. They've got a paid-for home and the kids are through college. Now, Household B, age 50, $5 million invested. They need $300,000 a year. They've got an $800,000 mortgage and their 2 children are approaching college. So it's the same $5 million, but very different circumstances. Walk me through why these are totally different retirement plans.

Yeah, they are. I'm just thinking about couple number 2, man. Just, I feel for them. You're right. They should be concerned. You know, college costs, as you know, that not only are they expensive, they come at a time generally when couples are trying to do that 10-year sprint, you know, get all their financial affairs in order so they can retire and drift off to Arizona and live the life that they always dreamed. And so couple number 2 has an uphill climb. They have a lot of big expenses that happened, and they have a heavy lift. They don't have a lot of margin for error.

Right.

Couple number 1, because of those bigger expenses, are behind them, uh, and they're living a more cost-conscious lifestyle, it's much easier for the $5 million to support their lifestyle and their spending patterns. So their $5 million can be invested different. It isn't as heavy a lift. And so completely different circumstances, same dollar amount.

So it's interesting the way that you frame that and discuss that, Nolan, is that it almost seems like the number of your investable assets, that $5 million, is less consequential than that spending rate really is.

Yeah, you're right. The spending is the numerator, denominator is the asset. And if the numerator is low enough, then you're right, then the asset doesn't have to work as hard, the denominator or the total assets. So it is all about the spending.

Absolutely. And so let's talk a little bit more about spending. You know, people who are in this situation have this much money, they tend to be pretty good with money. But then there are also people on the other hand that make a ton of money and aren't great with their money. So at what spending rate would you be uncomfortable for someone who's only 50 that's talking about retiring?

The lower the number, the better.

Right.

The Mendoza line that always uses 5%. We've seen a lot of 2s and 3s. Yeah. So the lower the number, the better. I'm really comfortable at a number that's 3.

Okay.

4 could work. 5 is the top end I'd like to be.

Yeah. And starting from 50, you've got a long runway there too, as well. So that definitely has an impact on that percentage as well. And then how does having a mortgage or some future expense also change the calculation?

Yeah. So you can model that. I just did one this week with a client who has 2 kids and we model in a wedding for each of them. And then we also model in actually new cars every 7 years. So you can model in these expenses and build your model for them. But certainly it taxes your model a lot more. Um, the more spending you have, the higher that numerator, the more the model has to work.

Right.

So, uh, you really need to be careful. And you were mentioning earlier, Brian, you know, hey, age 50 is really young. I have 12 years before Social Security kicks in. And usually, even if I was lucky enough to work someplace that had a pension, most of those don't kick in until 62.

Right.

So I have all— and if you look, take a look at my restricted bucket, bucket number 1, where my 401s and my IRAs are, I'm restricted from those to age 59 and a half.

Right.

Now you layer on healthcare expenses, travel. You talked about the home. I may probably want to upgrade, but at the very least, well, I need to update and remodel at some point. If I have kids, you know, if I have kids and I'm 50, it's likely my kids aren't through with college yet.

Yep.

So, and when you look at those numbers and you model them over many, many years, you know, at 50, for the average healthy 50-year-old, the life expectancy for one or both of those couples is probably another 40 years. That means they're going to be in retirement for longer than they were working.

You're doing a great job of just teeing this thing up from one subject to the other, because we're going to talk about why 50 is a long retirement and what that matters to you if you're making this, trying to make this decision. Uh, in your early 50s. So there's a huge difference between retiring at 50 and 65. As you mentioned, at 50, you're not funding a 25, 30-year retirement. We're talking about 40-plus years. So one of the big things is that, that we've talked about in other videos is the sequence, sequence of returns risk. So as you mentioned, you're restricted in the types of money or the, the areas where you can pull money when you're 50, when you're retired. Talk about sequence of returns risk in plain English and why a bad market in your 50s could really crush your plan in this case.

I think if you're 50 and you're working, Brian, it isn't the death blow that it could be as if you're 50, you retired early, there's no more income going into investments, there's only money coming out of investments. Very different. And in that case, what we mean by sequence of returns, we mean that we know if you take a 3 to 5 year, but let's say 5 years in the market, All 5 years are likely not going to be positive years. We may have 1 or 2 down years. What comes to mind for me is 2022. It was the first time— it's only the, like, the 5th time in the last 100 where we had both stocks and bonds down double digits. It was a really bad year.

Yes.

So if I'm age 50 in 2022, I have stopped working, so I'm no longer contributing to investment plans. I'm only withdrawing. And 2022 January was my first year of withdrawals. I may start the year at a 4% withdrawal rate, but if my portfolio decreases 20%, that withdrawal rate kicks up a lot. And so it matters. That outcome is very different than if I've gone the first 4 years and my portfolio has increased.

Yeah.

And then year number 5, the portfolio goes down. That's the danger of the sequence of risk.

So let's talk about—

sequence of returns.

Sequence of returns risk. Exactly. So let's talk about someone 50 coming to us and say, okay, I've got $5 million, I'm retiring, figure it out.

Okay.

And so you just mentioned the sequence of returns risk, and that's a huge problem if you just maintain that growth allocation you've had your entire career. So talk to me about how reorienting in a way that you have more income coming in from income securities more reliably, how that might assuage some of that sequence of returns risk.

So this would be a case where you could do this differently. You know, in an earlier video we talked about this 60/40 portfolio. And so if I have $5 million and I'm 50, I could put $3 million in growth shares I think are gonna go up. I can put $2 million in income securities that are paying me 6%. We talk about the $10,000 a month. But what if my spending's $15,000 a month?

Yeah.

Or $20,000 a month? It's a different math, but we can solve for it differently. Instead of 40% of the portfolio being dedicated to income securities, we may need to go 80% and then have 20% our growth bucket. And what that means is I'm going to get more income today because the income securities are paying it to me, but I get less growth in the future.

Right.

That may work for a lot of people. That trade-off may work. It may work just fine for a lot of people. There's a lot of ways to solve for it. It depends on facts, circumstances, and more importantly, Brian, psychology.

Yeah. And spending.

Yes.

Well, and it's the way you lay it out, it does feel like a difficult problem to solve, right? Because you have at least 10 years before you can access any of your retirement funds. And so it leaves us as advisors, if, you know, you're set on doing it this way, it's a difficult problem to solve, honestly.

And think about this client. There's a lot of clients, Brian, that do want predictability and income and they don't care about growth. So think about annuity investors. So that's another sort of income security that we're going to look at as a tool in the toolkit. We can go to an annuity provider, which is just— think about it like a personal pension. It's not exactly that, but conceptually, let's think about it like a personal pension.

Right.

We have a lump sum of cash. We turn it over to an insurance company or financial institution, and they promise us a series of payments for life. So maybe the money's gone. But the income's guaranteed to us for forever. Now generally with those, you don't get a raise.

Right.

So I get the same amount, and if I'm getting $10,000 a month today, 10 years from now that $10,000 a month isn't going to feel the same, and it really doesn't have a chance to go up. But for some people, that is a good solution. They— for some people it works, and we're happy to provide that solution if From a stomach standpoint and a sleep-at-night standpoint, that's what works for the investor. So I think the takeaway here is facts and circumstances are one thing, but we're a custom shop, so we're able to marry that with what, what works for that client.

The thing that comes to mind as we discuss this, Nolan, is it's possible for most people to retire with $5 million at 50, but it's going to require some creative advisory and game planning around what you're going to do. Because if you just go in with the way you've been, just all the money in your 401 and growth stocks and all that different type of stuff, it's going to be difficult to achieve what you're trying to get to.

Yeah. And at least not in a smooth fashion. Let's put it that way.

Right, exactly. So, one thing that people talk about early retirement, lots of people, you know, we're in the business of people trying to retire or want to retire, but there's one thing that holds them back from early retirement in a lot of cases, and that's healthcare. So when you're 50, obviously, you've got 15 years before you have any opportunity to access Medicare. And then at that point, your healthcare expenses are more or less fixed. But if you retire at 50, you've got 15 years to fill in with healthcare as a bridge. And we've seen healthcare costs, health insurance costs go up and up, especially if you're in the individual market. not in an employer plan, we're talking $20,000 to $30,000 a year before Medicare for that 15 years, depending on your family size and whatever else is going on. So in that situation, how do you plan for this 15-year healthcare gap for this 50-year-old that wants to retire?

I mean, that's scary. That's probably one of the first things that anybody who wants to retire early I mean, that's the first thing you need to consider.

Yeah.

How am I going to get my healthcare covered? And then you need to model in the amount. So it's a really important first step. And from there, we also need to think about a mortgage because at some point that's going to be the next biggest cash outflow monthly, right, is going to be my mortgage. So it makes sense to think about, should I pay my mortgage off? When should I pay my mortgage off? But that along with healthcare, Brian, if you're retiring at 50, I think those are going to be at the top of the list for things we want to, we want to flush out.

Let me ask you this. In your experience for people who have retired early, do people tend to underestimate what the healthcare expense is going to be?

I think they know it's coming. They know it's coming. And so I think it can be scary. I had a client that Retired, moved out of state, and because he retired, his income number dropped very low. So he qualified for subsidies from the state. And so his first year or 2 in retirement, he wasn't paying— he's paying very little for healthcare.

Okay.

But then all of a sudden, you know, this year in the Reconciliation Act they did with Congress, right, um, those subsidies for most people kind of went away, and the healthcare premium went up for everybody on the exchange. I think the number was like Over 20% was the average increase. So in this case, this client was shocked. I mean, he knew the number was going to go up, but it was a shocking amount. And so, yes, there can be surprises there and you can model it, but, you know, you just really need to be careful with that number and make sure that it's comfortable because in some cases it may make sense to keep, you know, if the number is high enough.

Right.

It may make sense to keep working or to do something creative, like maybe going to part-time, right? Remote work. We can still keep your healthcare, but have a lot less stress.

So this is not a rounding error by any means. This healthcare expense is a huge factor in this whole decision here. Even if we use $25,000 a year as a middle-of-the-road calculation over 15 years, we're talking about hundreds of thousands, if not $1 million just spent on health insurance. And that can really make a huge difference on how much you've got when you kind of get to the actual full retirement age as well. On the other hand, let's talk about what a couple of extra years of working might buy you. So the math changes if you go from 50 to 53 or 55. And we've seen that as we run these retirement projections for people, just a couple of different year, couple more years working can really change the mathematics. Here's what I wanna push against the idea that your choice is just between Retire at 50 or retire at 65? There's something in between. So as a simple illustration, if you've got $5 million compounding at 5 years— 5%, I'm sorry, for 3 years, that $5 million becomes $5.8 million. And that's before we're talking about another 3 years of savings and income. So financially, talk about what another 3 years might be able to buy you if you move from 50 to 53.

Well, I know it can buy you some extra income. in retirement because I have 3 extra years, like you mentioned, where I'm not withdrawing. So my portfolio has 3 more years to grow without me taxing it. That means much more than 3 years on the back end. That means my probabilities of running out of money, they have decreased.

Yep.

And my ability to possibly spend more in retirement because I deferred and worked those extra years have also increased. And just think about it. The gap that I have from 50 to 59 and a half was roughly 10 years before. If I work 3 extra years, now the gap's only 7 years. I've cut my gap by 30%. That is a meaningful improvement on my outlook for retirement, retirement income. Yeah.

And if you work at a corporate job and you're at the higher echelons and you're getting stock compensation, so we're talking about more RSUs vesting, we're talking about more deferred comp. We're talking about maybe 3 more years of bonuses. And we're also talking about your Social Security economics when you actually get to that age, you know, 12 more quarters at a high level of income. Those are a lot of different factors that are going to change your overall retirement mathematics. Any thoughts around that?

No, you're exactly right. And as I'm thinking, you know, 3 years, not that long. We just had a colleague, Freddie, who just sent his younger daughter off to college. So she's going to graduate in probably 4 years. So it's shorter than a college education. And generally when you get around 50, you understand the shortness of time. So even as long as it's not a job you hate and a boss you hate, if you can endure, those 3 years can really do you well on the other side.

So what we're saying here, Nolan, is we're not saying don't retire. It might be an option for you, but We want you to understand what you're buying by staying a few more years. And maybe those 3 years are not worth it to you.

That's fine.

But we want to make sure that you know that you're making that trade-off deliberately and with, you know, eyes wide open of the consequences of doing that. So that's the idea behind 3 more years, what that might buy you. At least consider the math around, around that. So the last thing we want to talk about before we close this out, Nolan, and this is more of a lifestyle issue, this is a little bit less of a financial issue, but it's what I like to call the 100 to 0 problem. So a lot of times we talk when we ramp something up, it's 0 to 100. You know, how fast do you go 0 to 100? But when you're a hard-driving executive, and you must have been, or some, you know, business owner or what have you, you must have been to generate $5 million of a portfolio. People have spent 25 or 30 years building a career, managing people, solving problems. The calendar is full. You know, you've got a full life and, and A lot of meaning and a lot of things going on. Then Friday, you're an executive, you retire at 50. Monday, you're retired and you got nothing going on. Nobody's calling anymore. You don't have the same purpose. That is a dramatic transition. So talk to me about what you've seen in this transition. Does it work well for people? And what are some of the pitfalls that you've seen in this going from kind of from 100 to 0, rather?

This is a really interesting topic, Brian, because it's been coming up more and more. And as I mentioned to you, if you're in corporate America like most of our clients are, at some publicly traded company where they get company stock, and they're generally, like you mentioned, successful manager, VP, director level, usually somewhere between 50 and 60, there's a good chance that some package is going to come your way. They're going to be getting rid of a division.

Mm-hmm.

there'll be an M&A, you have a bad boss, they're eliminating positions, whatever. And so this comes up, this, hey, should I retire? First, it's always, can I retire, right?

Sure.

It's the numbers. And, uh, fortunately, a lot of our clients, the answer is yes. If you just run the model, should be no unfunded years. The numbers look great. The next question, they don't generally ask this one, but I think what is implied is, Should I retire?

Right.

And so there's where I think it gets pretty interesting. I think people need to plan for retirement. I like the idea of mini retirements. Try it out before you do it.

Yeah.

You know, if you think you want to live in the desert or in the mountains in Carolinas, yeah, spend 6 weeks there. Go on successive 2 or 3 week vacations. Try it out. If you want to sail the Caribbean in a sailboat, take a couple-week vacation, do it. If you're not sure exactly which state you want to live in, use a few quarters and visit each one. Look at house prices, stay there for a while, do an Airbnb, see if you like it. If you think you can retire and be a golf instructor, start doing it on the weekends. I like the idea of the mini retirement so you have something to transition to. Far too often we see somebody that just says, you know, basically there's not a lot of plan for a lot of people on the other side. It's, I just don't want to do this anymore.

Yeah.

And so retirement sounds like the magic out button. It's magic through that door. If I can get through that door, all the great things are on the other side. And often, far too often, we go through the magic door, we get to the other side, And the room is empty. We have to create the new reality. And Brian, unfortunately, that's where we've seen some people get stuck.

Okay, let's talk more about that. So, talk about, you know, have you seen people who get the finances right, but the retirement part, the life part, not so much right?

That's far more often what we see than the other way around, at least for the clients that we work with. Yeah. And It takes sometimes, I mean, there's an adjustment because it is a Jane Doe 2.0. You really have to recreate yourself. You mentioned earlier, when you're in corporate America, you're needed, you're wanted, you're valued, you're desired, you have a place to be.

Yeah.

And all that goes away when you turn in the swipe card for work. And now you have to recreate that. And a lot of times you're not gonna recreate that. But you still have to have an active and full life. And a lot of our clients are hard-charging professionals. So a lot of them haven't developed like a balanced life with hobbies and a network outside of work. And they don't have a balanced and full life, which is like you mentioned, which is probably how they got— became successful to begin with.

Exactly.

So now I'm on the journey to build a new life and become a new person with new activities, A new circle, a new daily routine. And we can underestimate how big a lift that can be.

I think people need to consider the value that they bring to the world, right? And, you know, maybe the choices to stop doing what they're doing in the corporate world. But, you know, talk about some of the things that you've seen have been successful when people have made this, this leap to do the early retirement. What are some of the things that they've done right to get it right?

So I've seen a lot of people become active Uh, in other endeavors. Some— I've seen some clients become active at the church, so they take on a volunteer role at the church, working at the church. I've seen a lot of nonprofits, some board memberships. We have one client that works for a children's organization. She loves it. Once or twice a week she works there. She's not in management, she's a worker bee. It's a low-stress activity. She loves it. Other clients work at the animal shelter. Volunteer their time working with animals. Consulting, we see a lot. We see a lot of people working at a local college to help out. Maybe it's a SCORE program, uh, maybe they're going into the classroom to work with young adults and young kids. So a variety of activity. I think the common theme is they're staying engaged somehow, even if you're on the association board for your condo building or your neighborhood.

Mm-hmm.

It's those skills that you've accumulated in your career, not only technical and professional, but those people and relationship skills. Keeping those put to use keeps the mind and keeps you sharp. As you know, Brian, things change quickly, especially with this onset of AI.

Yeah.

Yeah. So you may have great skills today, but if you don't keep them sharp, they're kind of gone tomorrow. And so, um, those are a range of things that I've seen. How about in your experience? What have you seen that have caught your eye as it relates to people in retirement?

Yeah, I think mentoring younger people is a big one. I think one of the things that's missing in corporate America, you talk about there's all these people who are packaged out and we're getting rid of the people who are in their 50s and 60s. Well, they've got a lot of wisdom to provide to younger, younger folks, you know, on the come up. And I think that's one of the things that's missing a lot in corporate America is we're pushing those people out that have the wisdom and experience. For instance, someone who's been a great sales professional for 30 years, you don't think their experience could be really useful for someone who's in their 20s and 30s running a sales organization? I think that considering the opportunities of how can you— is there a place that you can carve out where you can be a mentor to someone who's going to be in your position at some point soon? I think that's definitely something that's worthwhile consideration.

No, that's a good point. That is a really good point. I really like the example of the, uh, you mentioned the sales mentor. You know, one of the things I was thinking about that came to mind, like, if left brain ever forced me out, like, or if I retired early, what would I do? Um, I think, you know, I used to be a football coach.

Yeah.

Um, when my son was playing flag football. So I think that would be a great thing to do, uh, where you're being engaged, you're working with the kids, but it's also a great time. So, um, that would be another activity people can get involved with. I know maybe some people used to play tennis or golf, and that would be some other activities that you can do, not only as an avocation, but as a vocation as well.

Well, I think, you know, giving something back to somebody else, I think, is kind of what underlies everything we've mentioned here. But, but the bottom line here is, is that financial independence, it's, it's giving you the opportunity to walk away from your, from your job. But You got to think about, you don't really want to never work again when you're 50. You've got 40-plus years there. That's a long time. That's a long time.

Yeah. You were mentioning earlier about the brain drain, you know, all of these successful people and, you know, they, they leave and they have things to give back. You know, unfortunately, Brian, in corporate America today, far too many people just don't like their job.

Right.

You know, it's become stressful. The company heaps the work on them. Or somewhere down the line they got a bad boss. Maybe there was, there was M&A, and so now you got a new reporting structure and they're just like, ah, forget about it. So I hear that far too often, and that really is a lot of times, if it's not health, that's what motivates people to leave. They just don't— the culture's changed so much, they don't respect their boss or they don't like them. They're in a situation and they're just like, you know what, I'm just going to be done with it.

You know, it's interesting and it comes to mind. We haven't mentioned this yet, Nolan, but I think sometimes people just need a change.

Good point.

And maybe it's not retirement. Maybe retirement seems like the most obvious change, but maybe they just need a change or maybe they just need to go away for a couple of months and come back. Maybe they're just a little exhausted by what's going on. But it's a huge decision to just say, I'm done.

You know, the feeling that I get, Brian, is— I know it's going to sound weird when I say it. I just don't think 50 is old.

No.

50 just seems young. And you know what? I think 60 is young. I was back in Detroit visiting some family. My dad's— I was visiting my dad for his birthday and visited some friends and some relatives. And I met with my ex-mother-in-law who happens to be late 70s. 78 she is. She still gardens. I mean, I mean, you couldn't tell her from a 60-year-old. So I asked her, I said, Doc, I said, you know, I know your birth certificate says you're 78, but tell me, you know, do you actually feel 78? She's like, no, that's the crazy thing. No, I still feel good. I still feel young. I got energy. So 50 is extraordinarily young.

Yeah.

I think 60 is young. I mean, this is barring if you don't have health issues.

Yeah, yeah.

But for most people, we see far too many people. We have clients that are in their 70s, 80s, even still out and about traveling. And so there's a wide range of outcomes. But I don't think the ages in today's— the way the lifestyles are, medical science, technology. Yeah, I don't think 60s and 70s are a remand to the wheelchair anymore.

I think the bottom line is you want to make sure you don't just retire, but you win retirement. Is kind of what we're saying here. So let's close it out and say, you know, the question is, can you retire at 50 with $5 million? Maybe. But, you know, it really comes to mind, it's really not the question. The questions are, what do you spend? What obligations are ahead of you? How much do, how long does that money need to work for you? What does healthcare look like? And what do you want your life to look like? Bottom line, that's, we spent the most time today on that, when work becomes optional. So if you've built significant wealth, you're trying to figure out when work can become a choice. Rather than a requirement. That's exactly the type of planning we do here at Left Brain. There's a link to schedule 30 minutes below with our team. Nolan, any final thoughts?

Yeah, I just think when you're going from 100 to 0, really what you have is now you have control of your time. Yeah. And now I have control of my time. What does my life 2.0— what do I want that to look like? The $5 million just helps support that.

That's right.

But the big work is on me to recreate my 2.0.

And if, you know, if you're considering early retirement or you've got friends that are considering early retirement, like and share the video with those colleagues. Hopefully, we can help out with what we discussed here today. And, you know, hit the bell for notifications and comment down below. What are you planning to do in retirement? We'd love to interact with you and hear how you either, if you're retired, how you've won retirement or what you're planning to do when you do retire. Uh, this has been, uh, the Left Brain Investment Committee meeting. We're happy to see you again next time.

Thank you.