Walter Willett on Controversial Low-Carb vs Low-Fat Study

In fact, it's worse than useless because it's misleading. This is so fundamental that there, probably a retraction would be the best way to do it.

This video that I'm going to cut to momentarily is going to be a conversation between myself and Professor Walter Willett, a name I think a lot of you will know. It's from a longer conversation that was hosted by Chris McCaskill at Plant Chompers on YouTube, a good friend of mine. He was nice enough to fly over from his home in sunny California to meet us in Boston at Professor Willett's home. And one thing we talked about was our recent reanalysis of a metabolic ward trial, a study that was published in 2021 by the Hall Lab looking at low-fat versus low-carb diets. I'm going to let Professor Willett's words speak for themselves. You might be surprised to see what he has to say. Let's get into this reanalysis. I'll describe it a little bit, and if you need any clarification— I'm pretty sure you've seen the paper and reviewed it quite in detail, but if you think that the readers need any clarification, we can go into that. So there was a 2021 study published in Nature Medicine, I think it was February 2021, in which a team at the NIH wanted to compare a low-fat diet to a low-carb diet with the primary outcome being energy intake. The idea is whatever diet caused people to eat less, you know, they'd lose more weight, and that would be good for long-term weight management. So they kept these 20 people in a metabolic ward and had each do both diets, both low-fat and low-carb, in a randomized crossover manner. So 10 did low-fat then low-carb, 10 did low-carb then low-fat. And the analysis was to say, you know, to look at, again, again, the primary outcome, who ate less. And what they found was people who were on the low-fat diet, or when people were on the low-fat diet, I should say, they ate fewer calories and had more weight loss than when they were on the low-carb diet. And for a couple years, that has stood as this is showing that, you know, low-fat might be better, better than low-carb for weight loss or reduced caloric intake. And it was seen as, they say in the paper, disproving the carbohydrate-insulin model of obesity. Now, as I said before, this had a really, a lot of nice bells and whistles. It's a metabolic ward randomized controlled crossover experiment. You know, very rigorous and well controlled. And so, and it's published in a top-tier journal. So we're inclined to just believe what the results of that are. And it stood for a, couple years until, well, you know, the data were looked at again. And there was one line in the paper, which I'm sure you noticed, in the methodology where they said something that was a very important claim, even if only one sentence, which was, "We analyzed for diet carryover effects and found none," or they weren't significant. So, you know, you know what that is, but I'll explain to those listening. That, oh, do you want to? I was gonna say, diet carryover effects, as the name suggests, is how, you know, one diet, say the first diet, will affect the intervention that follows. Because there are adaptations to dietary shifts, particularly with macronutrients, there can be an effect whereby the one bleeds into the other, the first bleeds into the second, which is usually why in these kind of experiments you'd ideally want a washout period, you know, so you can reset. So to speak. The interesting thing about this study is they didn't include a washout period. So one diet ran directly into the second. And upon reanalysis, despite the original paper claiming they looked for and did not find diet carryover effects, which they provided no tests to substantiate, which is also very important and kind of alarming, we found massive diet carryover effects that were actually larger in magnitude than the main effects. Found. So I guess my first question to you is, do you feel this invalidates the results of the original trial?

Because of this really deep flaw in first not allowing for the potential of a carryover, a crossover effect, that really was a serious question to begin with. And it is very disturbing that In fact, there was a massive carryover effect and they somehow overlooked it, ignored it, didn't analyze the data correctly in their original publication. But I think fundamentally this does really invalidate their conclusions.

Yeah, one of the comments as to try to explain this statement, the "we looked for and didn't find diet carryover effects," was trying to parse between-group and within-group diet carryover effects, which to me, I'll be very blunt, seems like a distractor. If you're doing a, you know, randomized crossover trial in which you have two diets, you do between-group analysis. This is just what you do. And then the whole idea of what even is a within-subject carryover effect analysis? Because if you want to look within subject, each individual subject, correct me if I'm wrong, but you need to look at the diet carryover effect of low-fat to low-carb versus low-carb to low-fat within subject. But if each subject only did one diet order, then there's nothing to compare it to. It's like me telling you, "Professor Willett, you know, I have a lot more energy when I drink coffee after a nap as opposed to having a nap after I drink coffee," if I'd never ever in my life had a nap after drinking coffee. It's not a claim that you can really substantiate. So what is— the logic behind within-subject diet carryover effects, or is it nonsensical?

I find this nonsensical. It doesn't— because really you are looking at the collection of the individuals, so what happens to one person is not really informative. So it's a little bit of a mystery to me what they're talking about there. It seems pretty clearly, if you do the right analysis, there was a massive carryover effect, right?

And then another thing that's been said in response to our reanalysis is that, "Oh, it doesn't matter because everybody on the low-fat diet ate less than everybody on the low-carb diet," which to me, it just misses the whole point of the reanalysis because the low-fat diet was effectively stealing credit from the low-carb. I'm interjecting here for just a moment. I think the topic that we talk about is best explained through an animation. This is a graph taken from the NIH lab that produced the original 2021 Nature Medicine study. I want you to notice a few things. First, red signifies the low-carb diet first, whereas blue signifies the low-fat diet first. You can also see circles and squares. Circles represent the low-carb diet and squares represent the low-fat diet. The first thing to notice is quite obvious. At the 2-week point, at 14 days, there is a large shift when transitioning diets, whereby people on the low-carb diet first— red— see a large decrease in energy intake at the 2-week mark, versus those who started on the low-fat diet see a large increase. Another thing that's important to notice is when you combine all the low-carb diet and circles, and all the low-fat diet squares, you see a certain distance, a delta between them. That's the difference in energy intake. And what you notice is that the difference between the squares and the circles, the low-fat diet and the low-carb diet, is much smaller than the distance between the red and the blue, the low-carb diet first and the low-fat diet first. This is showing very clearly that diet order makes a larger impact than diet itself in this study, which is consistent with our conclusions that the diet order effect dominates over and invalidates the diet effect originally reported. Another very important thing I want you to notice is if you take away the second 2 weeks, which is important to do because the second 2 weeks were biased by the first 2 weeks, then what you're left with is what is basically a short-term parallel arm study, but without diets preceding it to bias the effect. And what you see very clearly is that the low-carb diet, the circles in the first 2 weeks, are trending below the low-fat diet. In fact, in the second week of the first 2 weeks, after there's been some time to adaptation to a respective diet, you see that the low-carb diet is actually eating fewer calories in the low-fat diet, opposite to the original report. It's been a difficult thing in me trying to explain, especially just in social media spaces. It doesn't matter what the, the sum of the arms was at the end. It doesn't matter if every single participant on low-fat ate less. The study results are still misleading and invalidating. But you can see why that's such a compelling narrative to those who want to hang on to it. Oh, If everybody on low-fat ate less, then whatever you're doing now doesn't matter and I'm not going to listen.

Right. It is a little bit of a complexity to explain, but I think the important point here is that we have lots of evidence from earlier studies that people do adapt to the macronutrient composition of the diet, you could say the ratio of fat to calories, and it's likely that that adaptation will take a few days or sometimes a few weeks. And that adaption won't go away immediately either. So I think just the logic here, knowing that there is an adaption method and it takes a few weeks, and if you don't allow a time for that adaption to occur, your second 2 weeks of a study like this could be just totally misleading.

Yeah. So, um, this might be a little bit of a provocative question. I'm not sure if will want it redacted after the fact, but do you think the initial trial should be retracted?

Well, that's an interesting question. There certainly should be some red flags attached to the trial in the literature. We go to— most of us go to PubMed to look at papers and find them, and there's sometimes, if there really is a fundamental flaw, that they may be retracted and there'll be some comment about that there. Some— there's correspondence, letters to the editor, sometimes people look at this, but to me this is so fundamental that there— probably a retraction would be the best way to do it or sometimes people are allowed, authors are allowed to retract their paper, reanalyze it, and republish the paper. There's And this is a good process. It's part of the scientific correction method. And sometimes these are totally honest mistakes. It doesn't mean that people are corrupted or evil or anything like that. It's human to make mistakes. And this is a mechanism for doing that. If we just allowed the letters to the editor to make corrections, that's likely to be inadequate because a lot of people, myself, don't always look to see what the correspondence after publication says. So I think there does need to be some sort of flag here.

Yeah. And we already kind of alluded to it, but— or you just did— but this isn't just about reanalyzing one paper and correcting the record on one issue. It's about much more than that and also where we direct resources and how we design future studies. So I think you already said it, but if you wouldn't mind reinforcing, what are the lessons learned from doing a reanalysis like that and how we design future studies?

Yeah, hopefully we'll learn something about the general approach to doing studies like this, but in some ways we actually learned that 20, 25 years ago. When we looked at studies of low-fat diets and we found that in randomized studies there that lasted 2 or 3 months, people who were put on a low-fat diet lost a bit more weight than a few kilograms, sometimes just 1 kilogram, or that's maybe 2 to 4 pounds of weight loss on the low-fat diet compared to the usual diet or higher-fat diet. But that weight loss on the low-fat diet wasn't replicated when we looked at studies that lasted for a year or more. So we already knew that we should not be doing studies lasting 4 or 5 months, but we should be doing longer-term studies. So I think the investment in this kind of study is just pretty useless. In fact, it's worse than useless because it's misleading. And I think it's very troublesome to even do this study apart from the crossover flaw in it. But I think just adding a good washout period would not be enough here. You really need long-term studies, not studies of 2 weeks.

I agree.