Paper Mills, Peer Review Problems, P-Hacking & Everything Else BROKEN About Modern Science & Research Journals, With Emily Kaplan

My name is Ben Greenfield, and on this episode of the Boundless Life Podcast.

So they basically exposed that this work had been completely manipulated. All of the work from, let's say, 2006 to 2022 that we have relied on for Alzheimer's was based on this study published in Nature. Now, what's interesting is that—

Welcome to the Boundless Life with me, your host, Ben Greenfield. I'm a personal trainer, exercise physiologist, and nutritionist, and I'm passionate about helping you discover unparalleled levels of health, fitness, longevity, and beyond.

My friend Brent Bolthouse, super cool dude. I've known him for a few years down in LA. He always has these unicorns and usually people in the health or fitness industry who he introduces me to. And, he recently texted me and he's like, "You got to meet Emily Kaplan. She's doing these incredible things with what's called the broken—" Science Initiative. And of course, since I trust Brandt, I went and sleuthed around and watched some videos and visited the Broken Science website. And I actually kind of like what's going on. Very cool. And I'm not going to try to explain it because I've got Emily on the podcast, so she gets to fill us in on what all of this is because I think you could make a case that science is broken to a certain extent, at least the way that it's practiced. And Emily has some cool solutions and also some kind of shocking truths. So, BenGreenfieldLife.com/BrokenScience is where the shownotes are going to be, BenGreenfieldLife.com/BrokenScience. And, I forget, what's the URL of your website, Emily? It's BrokenScience.org. Oh, that's easy. Okay, BrokenScience.org. So, I just want to jump right in. What is it? What is Broken Science Initiative and This is your sneaky way to tell your backstory too because I want to know how you got involved with it.

So, I mean, the Broken Science Initiative is technically a publishing company or an education company. And it's Greg Glassman, who is the founder of CrossFit, who sold in 2020, and I partnered together on this initiative. I had been an investigative journalist and I've launched and sold a couple of different companies, but we had both been really looking at health and metabolic health, nutrition, exercise, I had spent a lot of time looking at women's health, and we'd both be— we'd become friends basically in that space. And we realized that through COVID, there was a lot of awakening and awareness about things not quite being what people were saying or not knowing who to trust. And we sort of traveled around and met with a lot of people and heard from some, you know, really well-established scientists. This is the first time in history that scientists are being maligned for doing science. And Greg and I were both like, no, it's not. Like, this has been going on in nutrition and medicine has been, you know, a mess for decades. And so we sort of put our heads together and spent about 2 years going back and reading a lot of the philosophy of science because we knew it wasn't COVID and we knew it wasn't just medicine. And we knew that we had things like John Ioannidis, you know, paper, which is the most cited research paper, which is that most replication— it's the replication crisis. It's most research findings are false. That was like 2005. But it was sort of like we, we were upon this moment where people were really interested in this, and we wanted to go back and sort of figure out like, how did we get to this place where corruption is so easy? Where, you know, you can have these peer-review, high-impact journals publishing research that we know actually doesn't have much clinical significance, meaning you may have some finding, but when the doctor goes and tries to implement it or the drug is given to millions of people, it's is not as effective as it appeared to be in the research. And so, you know, this is like a very long, in-depth conversation, which is really what Broken Science was founded to look at as step one. And what we realized was that people like Karl Popper, who sort of is like this godfather of the philosophy of science, had really started to question what is science and what isn't science. And I think, you know, Greg really had— part of his claim to fame is that he defined fitness in measurable terms, right? So, looking at power output and how do you measure that and how do you increase that? And so, when we started looking at science, we realized like there's no real definition of science. People will often refer to the scientific method or, you know, other measurement like p-values as a statistical tool to show validation. It doesn't actually do that. So, we can definitely like get into the weeds on that, but Karl Popper was sort of the first person. I mean, David Hume in philosophy did this before where he questioned induction. But Karl Popper really does this thing where he denies induction and he bases science on the ability to falsify results. And it's really an interesting story. He was very interested in the sort of advent of psychology. And in my research, I found that his family was actually really good friends with Freud's family. And Popper spent his early years really looking at working in a psychiatric hospital and being really interested in like, if I say my mother is the root cause of my depression, Can we prove that? Can we disprove that? Well, we can't really prove it, but we could maybe disprove it, or maybe not. So you look at Freud and everything is, it's your mom's problem. And I may say, like, my mom treated me horribly. My mom didn't, I had a wonderful mother. But like, I could say, it's all my mom. And you can't say like, well, no, Em, like, I was actually treated much worse than you and I'm not depressed. And so this notion of falsification enters our zeitgeist and that ends up coming out to be this sort of deductivist path, which really leads to the development of these statistical tools that we're relying on in peer review. So, if you go and you move forward to the sort of peer review process, we have this really sort of consensus-based way of deciding these things using these statistical tools.

Real quick, before you describe the peer review process, are you saying that you're okay with the definition of science being the ability to falsify results? Are you saying what you're about to get into is built upon a faulty premise. The whole thing is faulty. That's like a really big claim.

It's a huge claim. It's a huge claim. And I think like being skeptical of your results, trying to look at your work and see how you can disprove it, is certainly a part of the scientific process, but it doesn't define science. So our notion is that science is really defined based on predictable, like predictive value, or, you know, looking at probability theory. So, I mean, if we just take like sort of the modern idea that you have to falsify your results, there's lots of things I can falsify. This doesn't make it scientific. What makes it scientific is I make a bold claim about something and I'm able to show that I can do it and that I can do it again. And actually, that doesn't really take into account anything like peer review or even this null hypothesis significance testing that we're relying on. And so, like, take an example of like, I build a rocket and I say, I'm going to hit this target and I hit the target and I don't involve anybody else, right? Like all of the top secret work that we do doesn't ever go through a peer review process, but it's predictable and the outcome can be measured with probability. So, there's something called Bayes' theorem that we really think is the way that science should be done is using Bayes where you can take prior information And you can update your sort of hypothesis or whatever as you get new information and you get a better and better predictive value from it.

Is the reason that the peer review process, which I actually want you to explain, I know you were about to do that before I derailed you, but is the reason for the peer review process so that we could kind of sanity check the way that we've set up the ability to test our probabilities against a number of different experts? That's the only thing I'm thinking about as you're describing, well, the rocket got to where it was supposed to go based on probabilistic outcomes. But, isn't part of peer review the ability for other people to identify any potential flaws in the experiment that you ran?

I think it should be. And, I think as a gold standard, I think if you go back to pre-World War II German research, which they were actually doing incredible things looking at the metabolism. Like, there were— if you read those journal articles, right, the original sort of foundation of, you know, scientific literature and how do you work as a community of scientists to better each other, they're like hundreds of pages. And the scientists go into like every argument that anybody is going to put against them and they share all their data. We're at a point now where like you can't— most of the information you really want to know in a peer-reviewed journal article is in the supplement. A lot of that is locked away, right? So if you look at the sort of high-impact journals, it's like one of the best business models I've ever seen in my life, right? We are paying as taxpayers for those academic institutions to run. Most of the research is happening there. They then get a grant from the government to do the work in their lab that we've paid for. So we're paying again, right? The scientists do the work, and then oftentimes they have to pay for their submission. So as a former investigative journalist, I always like to say, like, if I write a story for The New York Times and I pay them, that's no longer journalism, that's advertising, right?

Well, the scientist is paying them, but really indirectly what you're saying is that we're paying them because through grants and taxes we're paying the scientists.

Well, multiple times, right? So like from the peer review standpoint, like, I think it's a beautiful business model because like they're— and then you got the editors who are peer reviewers are anonymous. So it's very different than journalism in that way too. And they're working for free. And then the article comes out and it goes in a publication that's thousands of dollars to subscribe to.

Wait, wait, wait. Who's working for free? The peer reviewers? Yeah.

Oh, so they don't get paid for that work.

They're not like on the payroll of the journal.

The peer reviewers? No, it's all— it's free labor across the board.

Okay, okay, we kind of sort of maybe put the cart ahead of the horse a little bit. You might have to describe in a little bit more detail what a peer review actually is.

Yeah, so usually as a scientist, you would submit a paper, right? You've done this research in your lab, you've written it all up, and you send it around, and you want it to be in this— the high-impact journals are the ones that have— hold the most weight, right? So Nature, the British Medical Journal, the New England Journal of Medicine, like, these are sort of the best places that you want your work to be published. And so you submit it, and then a group of peer reviewers— peer meaning are other people who are in a similar field, so they're theoretically qualified to evaluate your work— are supposed to go through and decide whether it's accepted for submission or not. That's sort of the job of an editor, right? If you're a freelance writer, you're writing something, you send it around to different publications, you hope somebody accepts it. They have editors that look at it. But they also have fact-checkers. There's no fact-checking. So when you get an article as a submission and you're the peer reviewer, you're told to assume that the work is done and that there's been no scientific misconduct or fraud. The datasets now are so involved that, especially when you start taking in AI, that, that it's really impossible to expect a volunteer to rerun any of your math or your datasets or your stats. So you're really taking a snapshot view of like, is this good or not? And good is often novel, meaning new. So they're not looking at replicated results and they're not really looking at things that we have known to be true or that there's a slight difference of. So it's— and again, like I would say, that's not very scientific, right? You're only trying to publish like new, exciting things. So that has led to a lot of problems.

The whole peer review thing is fascinating to me. So, a few quick questions about it. Peer reviewers are working for free, they're reviewing these articles that are supposed to be published in these journals. How do they get that job in the first place? Is this reading the classifieds ad over coffee and a donut and you see peer reviewer needed or how do they get the job?

If you think of it, I mean, this is another one of these areas where there's bias and potential problems. So like, if again, like to compare it to journalism, because I think that's what a lot of people think it is, right? Like good journalism, you have fact checkers, you have editors that are not associated with you at all. In this realm, it's like the journal goes out and they try to find other people. So like, let's say it's neuroscience, they're going to try and find other people in the academic neuroscience community who will understand your work and be able to, you know, tell them, should we run this or not? So that also means that like if you're peer reviewing my stuff, we probably know each other. We have probably worked together.

Well, why would I even want to do it in the first place if I'm not getting paid? Is it— does this like good for the resume to say that you were a peer reviewer for certain journals?

No, it's much more of a I scratch your back, you scratch mine. So I will have a journal article that I will want to submit and it will need to be peer reviewed by my colleagues. That's even worse. Yeah. So we have these massive problems with, you know, something that probably started with good intentions. I mean, like collaboration is super important in these fields, right? And being able to share your results and learn from each other and grow off of them. But what I am, you know, sort of— and Greg, we're like pretty obsessed with is how these things have gone wrong. And so when you start looking at things like, you know, most research findings are false, or like I have this talk that I've been giving around the country to doctors mostly. We have a medical society within Broken Science, and We also have Medfix, which we'll get to. So I sort of do these events where it's both groups, and I have a laundry list of editors from The Lancet, from the New England Journal of Medicine, from the British Medical Journal, who are basically saying, like, I can no longer stand by what we're publishing. So I actually have these quotes in front of me, 'cause I feel like sometimes when I say this stuff, people think it's so unbelievable. You have the editors of these major high-impact journals calling foul on their own industry, and nothing's changing. And no one's listening to them.

So, what you're saying, what I've gathered so far to summarize is that people doing research can pay for their research technically to be published in a journal which could introduce some bias. And then, the people reviewing that research to determine whether or not it should be published could just be their cronies and their friends who are scratching their backs? Yeah.

I mean, I think it's far worse than that. We have something now called paper mills. Where there are papers being published that you basically are just paying to have your name added to. And sometimes the work has never even been done. Like, somebody tracked down some of these labs and found the labs don't even exist.

Give me an example. So, paper—

if anybody, like, Googles paper mills, you'll find they're like— it's a massive problem in academia. And part of the problem is that you don't get tenure as a professor unless you publish a lot. So there's all this pressure to publish. So people are going around and they're saying, okay, I'm like, if you pay me $2,000, I'll add your name to this paper. We're pretty sure it's going to run in Nature. No, I get 2 papers published in Nature, the likelihood that I get tenure is way higher. So I mean, it's like wherever there's an opportunity for corruption, right? You can't rely on just like moral high ground that everyone's going to do the right thing. And I always like to give the caveat that like, I don't think people who go into science are criminals. But, I think we've created this sort of toxic pressure system for people working in labs in academia where they're not making very much money.

So, somebody could go on a podcast and say, for example, "I've had 3 articles or pieces of research published in the prestigious journal Nature or whatever," and technically, theoretically, they could have just paid to have their name added to that?

I mean, we just had this big scandal that— I mean, this is the tip of the iceberg really. You look at things like Dana-Farber, I'm in Boston, the preeminent Harvard teaching hospital here for cancer. Yeah. They had 53 papers flagged for image manipulation. So this is like another big problem. People are using Photoshop to change their images.

Wait, what would be an example of an image manipulation in a journal? 'Cause this is different than like, a dating website where you're trying to make yourself look 40 instead of 53.

It's very similar. So like, I want my research to show these cancer cells have shrunk.

Oh my gosh. So you're saying you could like go in with Photoshop or whatever and shrink the cancer cells?

So there's this site called PubPeer, which is amazing. It's all these image sleuths who basically are either AI experts or they're researchers and they're really sick of all the misconduct. And so, they go on and they have pseudonyms, and then sometimes, you know, there's a woman named Elizabeth Bik, who's been— The New York Times has written about, who's really good at this stuff. And they basically, as a hobby, for free, go through and look at all of these journals that have all these images, and they try to see which ones have been manipulated. Now, some of these you can see with the human eye. You don't need to have AI. But nobody, none of the peer reviewers are going through and looking at whether you use some tool to change your images. They're told to accept it that the research was done properly. So it's not— there's no checkpoint for that. So with Dana-Farber, there were 53 papers flagged by this British guy who I've talked to. He did everything that you, you would think you should do, right? He contacted the authors, he contacted the hospital, he contacted the journals that they were published in, and nobody did anything. Nobody would respond to him. So he wrote a blog post and that got the attention of the Harvard Crimson, which is the Harvard student newspaper. And so they started running stories on it. And then the mainstream media came in and became a little bit more interested in it. And Dana-Farber came out, and there's this guy who is the research integrity officer for Dana-Farber comes out and he basically says, like, we take this very seriously. We're doing an investigation, but we don't know that there's any intent. So that caught my attention because as I used to cover murder for 20/20 in primetime. And so I'm like, the intent is a criminal thing. It's not a scientific misconduct thing. And because our money is coming as taxpayers is funding this, why don't we have some federal level of oversight when somebody claims that scientific misconduct has happened? And so I looked up with my team the research into, you know, what does the statute say? And it's really clear, like it could just be negligence, which would mean that your name is on this paper and you didn't know what was happening in your lab. So intent has nothing to do with it. But by saying maybe this was an accident, most of the media just went away. We stayed on it and we kept asking them questions, and they won't even tell us if their investigation is ongoing, if, if the research integrity officer who's named on 3 of the papers is still running it. He reports to Lori Glimcher, who's the CEO of the hospital. She's named on some of these papers. So I have a whole presentation of the Dana-Farber scandal, but this is just one. So, there was also these guys, Begley and Ellis, who I don't know if you are familiar with them, but they realized that in cancer and hematology, you'd go to conferences or whatever and people would be like, "Oh," at the bar later, they'd say like, "Oh, don't trust Emily.

She's definitely doing tricky stuff." They literally said that about you?

No, no, no. I'm just hypothetically— some reason, right? They're at a medical conference and the presentations aren't happening. People are gossiping about who you can trust and who's doing good work and who isn't, right? Just probably like any conference you go to. And so they started to realize— Amgen is a big drug company— that a lot of their cancer drugs, like, weren't really very effective. And so they went to incredible length to try to replicate— I think it was 53 studies in cancer and hematology, hallmark studies— and they were only able to replicate 11 of them. Now, when I say, like, great lengths, they, like, rebuilt the labs. They went and got the original scientists to be involved. But in order to get the original scientists to be involved, they had to promise them anonymity. So they would say like, okay, Ben, like, we really, we're really struggling to replicate what the work you did. We need you to come and help us. We'll pay you to do it. And if it turns out we can't replicate these results because something was wrong, we will never tell anybody that your work couldn't be replicated. Now, what's crazy to me is the, you know, 40 whatever remaining studies that they couldn't replicate have never been retracted. So, this means that that work still stands, right? And, since we know people are building off of each other's theories or work, that means that I look at those 40+ studies and I think this is solid.

Well, the studies are still out there, but worse yet, I'm assuming the medications— That's right. —based on those studies out there are being sold.

That's right. But, I think when it goes to the moral authority argument, I'm like, "Why didn't any of those scientists feel a moral obligation to say, 'Oh, This isn't good. We can't replicate that work. I should remove it. So, the other thing that you and I talked a little bit about before was the Alzheimer's work, which was, I think it was a 2006 paper that was in Nature that basically was the foundational work on amyloid beta-56, which is the plaque theory of Alzheimer's.

Meaning that what I was taught, and I think a lot of people, is basically Alzheimer's is a result of plaque accumulation in the brain and research has shown that. Yes. And definitive.

Like, nobody argued with it. Everybody thought, like, this is great. And that was based on images. And so, in 2022, Science magazine did this incredible investigation into that work because they had heard it was, you know, not what they said. I don't know if I'm allowed to swear or not. I have a horrible mouth, so I'm going to try and keep it clean.

It's okay. I have cowbells. I've got clown horns.

Yeah, go for it. So, they basically exposed that this work had been completely manipulated. Right? So again, like Photoshop, changing the images to make it look like— Oh, no way.

Like the plaque was Photoshopped?

So all of the work from, let's say, 2006 to '22, or 2020, 2022, that we have relied on for Alzheimer's was based on this study published in Nature. Now, what's interesting is that Nature had a note after Science did this investigation, and it said, like, we're looking into this. We know there are big questions about this work, but they didn't retract it. They only retracted it this year because the lead author of the study came out and said, you're right, we lied, we manipulated those images, but the paper shouldn't be retracted because now subsequently we've actually done what we said we could do. And it's like, I'm sorry, like, how are you still allowed to be a scientist? You're admitting that you made up these images, you doctored the images.

This is important though, because you say he made up the images, but then Later on, research showed that the altered images would have been correct anyways.

Maybe. I mean, from the same person who lied about it in the first place, who's trying to cover her ass because now everybody knows that her work was fake. I mean, it's no good. You can't do that, right? Yeah. So we have this in cancer and hematology. We have it in Alzheimer's. And I mean, I think like, you know, to sort of move into what is broken science, we're really obsessed with these things. So, like, how did we get to this place where science isn't predictable, where we have billions of dollars going into these drugs that aren't actually clinically doing the things they say they can do? And how much, not just illness and sickness, does that cause? I mean, the average woman over the age of 65 is on 5 or more medications. And I always like to say, even if you believe that peer review is great and you think p-values are a measurement of statistical significance, can talk about why they're not. 5 or more medications, no one's tested that. So, they're not testing drug interactions. So, as soon as you enter another drug into this equation, it's the Wild West. We don't know. Ben Greenfield: Yeah.

I interviewed Joel Warsh yesterday. I don't know if you know about his book, Between a Shot and a Hard Place about vaccines. And I mean, I'm going to need an extra glass of wine this weekend after all these broken research podcast because he got it now. Basically, almost no vaccines have been tested against an inert placebo, but then he also made the point that you're kind of making that also none of them have been tested in a combinatorial fashion, right?

I mean, like, say what you will about vaccines, right? Like, it's such a hot button issue, but it's like you're, you're injecting an infant with multiple things that's never been tested in combination with each other.

Yeah, and we're not saying they're bad, we're saying like test them. We don't know.

I mean, I, I'm at a point where like, you know, my kids are vaccinated, but I— everything that I've read in last few years, it's just not been scientifically done. Yeah, you know, no placebo, like that's like gold standard clinical trial stuff, right? How do they get around that? And so I think, you know, from my perspective, and certainly I think why Greg and I felt like we needed to create this bigger organization was to help bring light to these issues that most people don't know, or it sounds like you're wearing a tinfoil hat. So like when I give this talk, I have a million slides, and I always say to people, you don't need to look at the slides, I'll share them with you. But I get that what I'm saying is super uncomfortable, and it's really hard to believe. And so, I will back it all up with everything that I use to come to these conclusions so that you can do the research on your own. And so, Broken Science is really trying to push to the forefront how we have this break, how broken things are, and that the individual really needs to take some ownership over their health. Now, I think one of the points to make about all the research that I've looked at is a lot of it is really focused in this realm of chronic disease. And when I say chronic disease, I mean like cancer, type 2 diabetes, Alzheimer's, mental health, and cardiovascular health, right? Well, probably obesity also. Well, I think obesity is a symptom of all of those, right? Yeah. Okay. So, if we say the medical system is going to treat you this way, like I go in, I have cancer, I'm going to be sent to the 5th floor of the hospital. I have Alzheimer's, I'm going to be sent to the basement. I'm depressed, I'm going to be sent to the garden, right? Like, they're not going to treat these things as the same problem. But what do we know? We know type 2 diabetes for sure is a metabolic dysfunction that has to do with insulin regulation, right? Yeah. And we know that the biggest risk factor for all of those other chronic diseases is type 2 diabetes. So, if you could treat type 2 diabetes, I mean, I would go even farther than to say that it's that simple. I think they're all metabolic dysfunction diseases. But, if you were to treat type 2 diabetes, you're getting rid of the biggest risk factor. So, enter Metvix.

I got to ask one more question about the research. I want to hear about Metvix.

We can go into all of it.

Okay. You're challenging a lot of dogmas here, so I have to ask you. You said that p-values are kind of a flawed form of statistical significance. Yes. Can you explain that part real quick because I don't want to gloss over that?

Yeah. So, the guy who came up with p-values was named Fisher. And he basically was like, "Please don't ever use these in medicine. This is like a tool for quality control." And what it does is it's basically, for people who don't know, the null is what you're testing against. So, you're supposed to be looking at an intervention versus the null. And the null just means none, right? So, it's earlier information you have or no change. And you're trying to look at, is there a change between these two things? Okay, sounds good, right? But you're not measuring that your intervention works or not, and you're assuming the null hypothesis is accurate, which is a massive assumption, right? You're taking some other piece of work that you've decided to compare it to. Yeah. And then on top of that, you're not actually— it has nothing to do with whether or not the intervention works. It's just looking at the relationship between these two data sets. And so it's— it— this is part of the reason why we have so much of this research that can't be replicated or that isn't quite right in terms of clinical significance because if you're using p-values, they're really easy to manipulate. You can change your sample size and get a significant p-value. So, they're just not a tool that's going to tell you whether your work can be replicated, whether the intervention works.

So, for a really simplistic example, let's say— let's go with something less morbid and serious than what we've been talking about so far. Let's say muscle gain, I want to show that full-body resistance training is better than just body part split training. And, if I really wanted to, I could prove statistical significance by choosing a sample size of 12 untrained overweight young males and then someone could actually run with that research and say, "Well, yeah, full-body resistance training is superior to body part split training because I have a very small sample size and not appropriate subjects." database? Is that what you're saying?

So it's actually easier if you increase the sample size to get a more significant p-value because you're trying to show a change, right? So you want there to be a difference. And I think what you're talking about is actually more of like cherry-picking your subjects, okay, to meet your basis, right? Whereas like you could do a really good sample match but you could still be stuck with this notion of like it's not a significant finding. And so, I wish I could think of a good example for you. There's papers that I've looked at in the last year where you can see in the data that there's an effect, right? You can see, let's say we take 12 people and we put them through rigorous resistance training, but what we're measuring is BMI. And so, we're measuring this thing and we're saying, "Oh, actually, there's no significant difference." But actually, if we had more people, And let's say the 12 people that had like a very small increase. I mean, let maybe— but maybe BMI is not the right example, but like let's say we did body fat, right? I mean, I hate BMI because I think it's such a bad indicator anyway. If we did body fat, like maybe their body fat decreases a little bit, but it's not enough to say it's significant, but it was decreasing, you know, across the board with all these people. You could look at the data and you could say, you know what, that's really interesting. It's not significant technically on the p-value or the t-test, But if we did it for another 2 months, maybe it would be right. Or if we increase the sample size to include more people, we would get, you know, people who are heavier. And so we would see a more profound effect. And so it has to do with like basically the number of people that you're including makes it easier for you to get a significant outcome.

So reaching statistical significance through p-value manipulation is something that is relatively easy to do and commonly done. Yeah, it's called pHacking and it's—

pHacking. Yeah. Okay. I'm learning new terms.

pHacking. What was the other one? Paper mills. You told me to Google. Pill. What was it? Paper mill.

Paper mill and pHacking. Okay. Yeah. And just, I mean, I'm sure you guys do with your show notes and stuff. If there's anything that you guys want me to send you as links out for people to look up this stuff, my goal is education. I mean, a lot of people want to have attacked me and are not comfortable with this stuff. And so I think it's really important to be able to be open to sharing, like, you know, all the sources. And the truth is, the sources are, are these high-impact journals. It's not like I'm finding this stuff off of like random blogs. Like, people are writing about it. It's just not getting covered. And then like, we can kind of get into the corruption, which is really, really lucrative. And so you've got somebody like me out on the road trying to talk about this stuff. Versus, you know, pharma and big food and all of these things that don't want this coming to light because they won't be able to control the research.

I interrupted you when you said enter Metfix.

Okay, so what's that? This is our— so we have the medical society, which was really about— it's about 50% healthcare workers and 50% patients, which is awesome. And so that has regular webinars where we have metabolic experts twice a month come on and do a Zoom call with everybody. So that you can do like sort of a deep dive into their research with them and then ask them questions. And then we have something called Journal Club where we take apart journal articles so you can learn, okay, this journal article everybody talks about as being great. What's wrong with it?

How do you do that? Is that like in written form? You do it on a video?

It's a call. It's like a video. Okay. So it's a forum kind of a thing. And, you know, I think our team is amazing. So Bob Kaplan, who is the head of research for us, was the head of research for Peter Attia. He launched The Drive with Peter, did all the research for Outlive, was the head of research for The Practice, is really clinical in terms of looking at all the research and being able to dive in deep and figure out what's right and what's wrong.

Are you guys related, Bob Kaplan and—

We're married. Okay. Got it. And then, I mean, I think basically what we've done with Metfix, so Greg sold CrossFit in 2020. It was bought by private equity. They're doing what private equity does so well, which is to really try to maximize profits, and they're not really willing to go out on a limb and say anything that's controversial, which is fine. I don't blame them for that. But, it created this real opportunity in the marketplace for us to come along and say, "We're willing to put a stake in the ground and say that you can reverse chronic disease in the gym, and we're going to teach you how to do it." So, we launched Metfix, which really stands for metabolic fix. And, we have a very rigorous training that people have to do over— it takes 2 days, but then there's a year-long class online. And, it really gets into the mechanisms of action of how does your body process food as energy and how does that either lead to health or lead to disease. And, if it's leading to disease, how do you reverse it? And so, we have about 100 locations in the U.S. already. We launched about 7 months ago, and we're now in the UK and Europe. So, it's really exciting because the stories coming out of those boxes is phenomenal. And so, the premise of the education is essentially that insulin is a storage hormone, that when insulin is high, your body's in this storage mode where it is trying to push as much as it can into the cells, right? And that's problematic if you're on a high-carb traditional, sort of standard American diet where you're consuming lots of sugar and carbs.

Problematic for what?

Well, because carbohydrates induce insulin, right? So, if your insulin is chronically high, you're going to be always in storage mode. You're not going to access the fat on your body for energy and you're going to be hungry and tired all the time because you are in this glycolytic pathway which basically requires that energy to be coming in all the time. So, what we say is the definition of health is really being metabolically flexible that you have these two engines. Right? You can be fat adapted and access the fat on your body, or you can be in a glycolytic pathway where you're accessing carbs and sugar as quick sources of energy. But, the problem becomes when you're in the glycolytic pathway for too long, chronically high levels of insulin, you cannot switch over. So, you can't access it. So, I always like to say because I think it's really important that people understand this, and I'm sure most of the people listening to your podcast are super fit, but when you see the average person on the street in America and they're obese or morbidly obese, and you say like that person can afford to skip a couple meals, you are so wrong and you are so ignorant in the sense of what that person is experiencing. They are exhausted all the time because they cannot use the fat on their body for energy as long as their insulin is high. So, lipolysis basically dictates like we're going to shut everything down and we're going to store it all away as long as insulin is high.

But, playing devil's advocate, didn't Kevin Hall's Metabolic Ward studies show that in an isocaloric scenario that that becomes much less important. In a hypercaloric scenario, the insulin piece does result in fat deposition and resistance to, I guess, lipolysis. But then, if you're calorie-reduced, it's kind of like— I guess the overarching message I got from his research was reducing calories trumps everything. High carb, low carb, you know, not even taken into consideration.

So I have a lot of problems with his research, which we can talk about. But I think one of the things when it comes to those arguments is that oftentimes when you're at a real calorie deficit, you're also eating low carb. So it's sort of hard to separate the two out.

Except for all those people now that are doing the sugar diet thing.

Yeah, well, that's— I mean, I think that's, that's really interesting. And I think like there probably is an interplay. I mean, one of the other things we talk about is seed oils. Yeah.

Well, actually, just real quick, the sugar diet thing, really, people say that, but I think what's the enzyme? FGS21, I think is the name of it. And basically, it's an increase in metabolic rate and fat protein in a low-protein environment regardless of whether it's a high-carb, low-protein or low-carb, low-protein.

I think that's right. And, I also think nobody has any idea what the long-term impact of that will be.

It sounds tasty though, gummy bears and gummy worms. I mean, sign me up if it were.

I'm going to let them try it for a couple more years and then I'll jump on I would feel horrible.

Just so you know, even though I'm questioning some of your hypotheses here, I'm very much a low-carb guy. I feel very good, very stable energy levels, metabolically healthy. Just for me as an n=1, it works really well.

Well, and that's the thing, right? So, when we go back to this idea of what is clinically significant, I think it's like, try it. I always say to people like, how do you feel when you eat a high-carb diet? Are you losing weight or do you have lots of energy? You don't, you just can't. And even athletes, I feel like there's this real stigma about like, I need carbs for performance. And it's really hard to convince people, like you'd actually— like Vinnie and Volek's work would suggest like, no, you probably don't. And there's a really cool study out of Sweden where they took these, I think they were cyclists, but they might have been long-distance runners. And they basically got them on keto. And then right before their competition, they gave them some carbs and it was like a super booster. So, I always say it's kind of like coffee. If I don't drink coffee for a couple of weeks, and then I have a cup of coffee, I'm like a whole new me.

Ben Greenfield: Sugar can become a sometimes drug. I was part of Volek's study, so I followed a pretty high-fat ketogenic diet for 12 months. I think since then, some research has, I think, disproven the idea that one could generate equivalent amounts of power and in some cases force in a glycogen-deprived state. But, I don't personally synonymize a low-carb diet with glycogen deprivation. I think that you can do kind of a cyclic low-carb approach, still approach any highly glycolytic activities with some form of sugar or carb intake a couple hours prior and kind of get the best of both worlds in terms of metabolic adaptations to a low-carb diet with occasional carb feedings when you really got to dig deep.

And, that's exactly what we say about metabolic flexibility, the ability to be able to switch back and forth as you need is optimal. Unfortunately, the average American is not in that state. And so, I think for what we're trying to do with Metfix, it's really to get the people in the gym who need it the most but aren't going in. I mean, we hear from a lot of people that like, "I need to get fit before I go." And, it's heartbreaking, but I get it. You look at the typical gym environment and you don't want to go in there and be the person who's 250 pounds and doesn't know any of the movement patterns and how do you cater to them? So, we have these classes that our Metfix affiliates are running that we call special population classes where it's just you're going out into your community and you're saying, "Hey, if you have type 2 diabetes, we'd really love to help you. Come in at noon. It's only people who have type 2 diabetes." Ben Greenfield: Yeah.

And this is similar to the CrossFit model where people can franchise this, start up a Metfix in their local city, for example.

Katy Kelly: Yeah. So, it's an affiliate model, not a franchise model because I think affiliate, we charge $2,000 a year to be a Metfix affiliate. You have to go through an application process. I mean, most of our affiliates are 10-plus-year gym owners. So, Greg likes to say this is like the PhD for coaches. But, we also have a lot of doctors taking our training because they were never taught any of this nutrition stuff. I mean, I always say that our curriculum is really controversial not because it's not based in biology and you can find anything we're teaching in a biology textbook, but because it is not taught to doctors and it is not taught to registered dietitians. Because once you learn this, you're not going to prescribe the dietary guidelines, you're just not. You're going to think there's no way. That has led to the chronic disease epidemic, not helped it.

Yeah. I want to go back to the journal thing for a second here. I'm just so pissed off about everything you told me for the first half hour of this interview because books, medications, magazines, there's so much based off of what comes out in these journal articles. I mean, half of the average health or fitness enthusiast or medical enthusiast blog feed is probably comprised of articles based on this research. How do you deal with that psychologically? Emily, do you have any research that you trust? Do you have any type of filter, any way of knowing if a headline based on research or if the research itself is trustworthy in the least?

So, I have some things that I've sort of developed from looking at this stuff for a long time. I mean, I think that's the reason we're doing Broken Science is because there needs to be a central hub where people can come and learn to make these assessments for themselves, but also learn what research is good and what isn't. So, we have this type 2 diabetes class online that people are like, "Oh, you should charge $2,000 for that." We're charging $150 because it's like I think everybody needs this information. And, it goes through the history of type 2 diabetes and it goes through all the research. And, it allows you to realize how did we come to these standard of care models that are about managing the illness rather than reversing it. We have one coming out on women's health that is literally my response to all of these perimenopause influencers who don't understand estrogen and insulin's interplay with each other. And, we have estrogen receptors in all of our organs. It's incredibly important to understand that regulatory system. So, that's going to break down all of them from a biological standpoint. But I also think there's things like, I did all these explainer videos on YouTube a couple of years ago for us that are looking at like, what are these statistical tools? Or, you know, relative risk versus absolute risk is a big one. So, like, I just did a little, like, mini video on— there was a headline that was going on everywhere that people were saying, like, if you live close to a golf course, You have 120% increase in risk for Parkinson's.

Because of like herbicides and pesticides, is that the argument?

Well, that's why everybody, you know, all of the people, I mean, I'm not in favor of, you know, Roundup, like, right, no thank you. But that's different than making a claim that's not right. And so, I think we lose the argument in some of these like sort of social movements, like some of the Maha stuff. I am, you know, definitely on board with all of that, but we've got to be accurate. Right? Because otherwise you lose all your credibility. So I went through that Parkinson's thing and just took it apart. It was basically like, anytime you see a headline that says 120% increase, it's likely that they're reporting relative risk, not absolute risk. And what you want to know is absolute risk. So when you rerun the numbers, it was like a 5% increase.

I'm just curious, did they isolate for the fact that a lot of people that live near golf courses— no offense if you're listening— are like old.

Right. Well, totally. I mean, no, they didn't. And also, I think like, for me, I was just looking at the statistical manipulation. So I wasn't trying to get into like the cohorts or like how they looked at that stuff. Or, I mean, like, you see this with nutrition stuff all the time, like the healthy user bias is huge. And so like, you're trying to limit your study to one variable. But like, let's say it's like people who work out 5 times a week are more likely to whatever. Well, they're also less likely to do a lot of other shitty things. And so, it's like, I'm not saying that the exercise isn't important, but we have this tendency to be so dramatic.

That's like the red meat ones. I think they call it the James Dean effect. James Dean, see the guy, like the bad rebel guy. Yeah, Rebel Without a Clue. Yeah. So, the James Dean effect where people think red meat is bad for you, you're more likely to also eat greasy French fries and a donut for breakfast and smoke a cigarette. And so, that kind of could skew the data a little bit about red meat being contributor to chronic disease.

Yeah. Well, red meat in particular has gotten such a bad rap, and I don't think it's fair at all. I mean, the World Health Organization, that colon cancer study, which was really what put red meat as a carcinogen on the map, they basically, they did food frequency questionnaires, which are notoriously bad. Imagine, I mean, you might have a perfect food log, but you're taking in people who have never been involved in anything health-related, and you're asking them to, like, sort of remember the food they ate last week and give you an accurate accounting of it.

I don't know what I ate on Monday.

We're talking on Wednesday. Okay, well, see, there we go. It's really hard to do. They also, when you looked into it, they quantified things like pizza was considered a red meat. And then they asked really confusing questions like, how many cups of ribs did you eat? Last month. Hmm. Ribs, like a bone in, bone out. How many ribs are in a cup? Like, I have no idea. It was so poorly done. And yet, it got all these amazing headlines. I mean, Women's Health, the Women's Health Initiative, a lot of people have been talking about this recently, but it's been known forever. They basically said that, like, you know, women who were on hormone replacement therapy were, you know, much more likely to develop cancer. That was a relative risk score. So let— and this— not— these numbers are not accurate, but like, let's say the number of women went from 3 to 6. So they said your risk doubles if you're on hormone replacement therapy, but it was out of 10,000 women. So actually what you want to know is 6 out of 10,000. That's your risk. It's not the comparative one. That's absolute— that's, that's relative risk. So I think like there's a lot of things like that that I think, you know, the average 8th grader could learn, but it's not taught. You know what I mean? Like, these are some of these— like, when you see something that sounds incredible, it probably is, right? And so, like, learn to look into those headlines. So I also always like to say, like, when somebody says, like, there was a significant result there, that's a p-value. So you have to remember that's not actually significant, but you're allowed to say it's significant if it is. Um, correlation and causation get mixed up all the time, right? So you'll see headlines in newspapers that say, like, red meat causes cancer. There's no clinical trial in any of that. It was a corollary study.

Right. Or honestly, like ApoB causes heart disease. It's another more confusing one for people. It's correlated. I've seen no research that it causes it. Yeah. There are some really good public science educators who I follow. They're not all male for any particular reason, but people like Alan Argon, Nick Norwitz, Lay Norton puts out some good stuff. Do you have people who you follow, who you trust, like icons, public science educators, people who you listen to them or read their stuff and you generally agree with it or consider them to not be a source of broken science?

I mean, I hate to say this, but I really haven't found anybody who I agree with on everything.

I don't want to call anybody out in particular. Yeah, I wouldn't say I even agree with anybody.

I just listen on Everything, you know, I think like it's just, it's really hard because it's like, I, there are people that I followed that I'm like, oh, this, this person's awesome. But then they do something and I'm like, that's not right, that's totally inaccurate. And then I, it makes me be like, ugh. I mean, I like, I feel like, you know, we all have to play the social media game, but it's really hard. I mean, it's hard to be producing videos all the time and it's really, I mean, like, I have an amazing, like, I think I have like one of the most talented teams out there And we're working all the time, but we're really more focused on educating our affiliate, like our MedFix affiliates and our medical society, because I feel like they will drive change in person in their communities. It's really hard to say like, oh, go listen to this person or this person's right. And I just, I mean, I'm skeptical, right? So I, it's hard for me when I hear somebody pronounce something that I've actually looked into and I'm like, no, that's not right, to then feel like they're reliable about things I don't know. Okay.

Well, let's talk about you then. You're an n=1. What are some of your non-negotiables? You've obviously looked into this stuff, you've seen a lot of the research on, I don't know, thermal stress and obviously we talked a little about macronutrients. I realize not everybody's Emily, but what are some of your non-negotiables based on what you've learned over the past few years?

Yeah. I mean, Bob is the number one n=1, just constant self-experimentation. So, You know, he— we have a sauna he lives in because the research behind saunas is actually really strong in terms of longevity and brain health and all this stuff. But I think really it's like, for me, the big rocks, right? Like, I don't think that there's a hack or a trick. I think it's like, think of the things that are going to drive the most impact based on where you are. So if the average American's main food consumption is sugar cereal and soda, like, cut that out, right? Like, that's a huge hack. But it's also like, we're, we're going to move the needle with things like that. I think once you become more fit, then you can start like really tinkering with things and playing with toys and playing with supplements and all of that. But I think for the average person, like just getting sugar out of your diet. So I have a huge sweet tooth, I love sugar, that I really do try to limit it. And I will do carnivore for like, you know, a month or so at a time to sort of feel like I'm resetting myself. And I've done that with alcohol too, where like you just get rid of alcohol for a month. And it's always for me like sort of a social experiment, because you realize like how many people are like, but Em, it's my birthday, right? Or like, what, can't you do it next month? And like, that's actually like a really good, good reset. But I also am like— and this is going to sound sort of cheesy, but like, I'm a big believer in play. And I think like, really, when you're playing and you're allowing yourself to be creative and have fun, you know, that often involves movement, right? So like, if you watch a child and they have to go do some mundane task, like clean their room, The kid who's really comfortable with themselves and in a good relationship with themselves will sing and dance and make a game out of that. So, I kind of like the idea of creating, like not gamifying health, but like making more play available. I think as grown-ups, we don't do that enough. And like, just allow yourself to be silly, allow yourself to have a break. I think that's a really great way of sort of like taking hard things and turning them into something that you can do. Because I think the truth, the best hack is honestly consistency. And it's also the hardest thing for people to do. So, if you can find creative ways of making those things that you don't want to do, like whether it's working out or eating right, more fun, I think you're going to be more successful.

Ben Greenfield: I think, by the way, I'll throw in there more fun or more productive. Like, my daily hour at the gym, you know, at 2x speed audiobooks is like, one of my primary forms of education. I consider it to be highly productive, way more productive than listening to music. And, that actually gets me into the gym. What am I going to learn today just as much as how am I going to move?

Do you wait to do that? You won't let yourself listen to the audiobooks anywhere else?

I save all the good stuff for the gym. I'll listen to stuff otherwise, but if there's something super juicy or a book that's highly recommended, that always gets saved for the gym. Yeah, I love that.

I think that's— you're incentivizing yourself to go. That's brilliant. Yeah, exactly. Yeah, or just think about you're giving yourself a treat while you're doing this thing that you need to do that maybe you didn't want to do today. I think the reward system is huge. And I think we overcomplicate these things to the point where then people quit. It becomes too much. They forgot their gear, they forgot they didn't go, and now they feel weird showing up for the class. Just give yourself a break, right? Everybody's kind of in this together. We all want to be healthy. I mean, this is the whole thing with like the sort of maha political stuff is I'm like, if you were to look at the base of like freedom pyramid, right, what do you have? You have health. We can't disagree on this stuff, right? Because if you are, if you care about freedom of speech or you care about freedom of assembly, great. None of that matters if you're lying in a hospital bed with tubes in your mouth, right? You're not gonna be able to talk and you're not moving anywhere. So we all have to agree that like we are failing. I mean, this country is, 60% of the country has a chronic disease. What are we doing about it? And I think it really has to do with undoing this corruption and then giving people a place where they can reclaim their autonomy as a healthy person. And so, I think any way you can get there is good. And I think people are different. So, like my workout routines vary all the time. Like I love rucking. I like being outside. I think outside is another one of these things that we really underestimate the power of. But, I also, I'll row, I'll do a constantly varied high-intensity functional exercise, which we call the Glassman Methodology at Metfix. I love those workouts because you just feel like a fresh slate afterwards, right? Like, you're out and none of your worries are in your mind, you're just trying to get through the next 10 minutes. So, those are really fun, and I love yoga. So, I do a lot of variation based on my crazy schedule, but also based on where I sort of feel like my body is and what do I need. And, I think you don't have to be dyed in the wool one thing or another. I feel stronger about the nutrition. I mean, I think that's why when we were building the Metfix curriculum, Greg and I were looking at what he had done at CrossFit. And, the CrossFit education is phenomenal, but it's really all movement. And, I said to him, "You always had nutrition at the base of the pyramid, but you never explained it to anybody. What if we just dive in on the nutrition piece and really give people the keys to the castle?" I know.

I explored your website. It's a little bit reminiscent of the old school WOD websites where you have your workout of the day, but then you guys have a recipe of the day. What do you have? Recipe of the day, workout of the day, then you have a learning, a mind snack of the day. It's kind of cool the way you have it laid out.

Yeah. So, we call that the Daily Fix. And so, it's a recipe that you should be able to make in less than 15 minutes, a workout of the day, and then a reading of the day because I really feel like it's should be the full body, right? And the thing I love about the Daily Fix is how many people write us and say, like, I'm cooking for the first time in my life, you know, or I haven't cooked for 15 years, you know, I've just been ordering out. And I really think it's very hard to reclaim health if you're not cooking. We haven't really talked about seed oils, but they're in everything and they basically are a, you know, they cause reactive oxygen species to be low. So people will say inflammation. That's another one of my like triggers is like when people say inflammation, it just means that they don't understand the root cause. Like, yeah, sure, there may be inflammation. Why is what you want to know. And so, with the seed oils, what we're seeing is that basically insulin opens the door to the cell and starts shoveling in all of the nutrients or sugar or whatever. And then, the cell is like, "I'm full," but it actually doesn't get the reactive oxygen species signaling molecule to tell it it's full. It tells it it's still hungry. So, when you eat these seed oils, your cells are saying, "Give me more, give me more, give me more." So then, the cell is spilling over, the amino acids are going to the blood, and you're ending up with this ectopic fat on your organs in places it's not supposed to be, which is really the beginning of chronic disease. And so, I think it's really the combination of the seed oils and the high-carb diet that makes it very, very hard for people to eat out. I mean, I was at a really fancy steakhouse recently, and I said to them, "I'm really trying to not eat canola oil." You don't use that stuff, do you? And they were like, "Oh, actually, yeah, we coat all the steaks in it before we grill them. So, we'll get you one that's plain." I mean, what? It's not.

Because it tastes so good. Yeah. And obviously, there's subtle nuances like has it been heated, has it been reheated, what temperature, what's the level of HNE and extraction methods, what's your current caloric status. There's a lot of subtle nuances, but painting with a broad brush because I tell this a lot of times to my clients. I'm like, eliminate seed oils when and where you can. And, I think one of the reasons that that works so well is it's really hard to eat a lot of ultra-processed foods and also to eat out in situations where you don't have a lot of control over calories and ingredients if you are not thinking about seed oils at all.

Well, one of the experiments that I like to tell people to do on yourself is there are these chips that are called Vandy and I should do a partnership with them because they talk about them all the time. They're great.

Actually, when they first launched, I'm like, "Oh, these are like crack." So, I actually put a bunch of money in Masa and Vandy when they launched.

Oh, well, I'd love to invest in them because it's amazing. So, for people who don't know, it's cooked beef tallow. And I feel like I can eat a bag of Doritos, no problem. I eat 5 of those and I'm full. It's a completely different experience. And it's why, it's because the saturated fat is making me full. The seed oils aren't.

Yeah. Well, we got to go pretty soon, but you know the ultimate fix for this is just combine retatrutide or tirzepatide with your seed oil intake. That will modulate the hunger signals, plug that hole, and then you can have your seed oils. It's reasonable. I'm not sure.

I'm not going to say yes or no to that one.

We need research on that. We need good peer-reviewed research on that. Emily Kaplan, brokenscience.com. Is the website. I'm assuming people can find out about Metfix there as well.

And, we also have metfix.org, which is its own property, but it links to—

it'll flip you back and forth. Yeah. Okay. Alright, cool. If you have questions, comments, feedback, if you want all the juiciness of the shownotes, go to BenGreenfieldLife.com/BrokenScience. Emily, thank you for educating me, pissing me off about journals, everything that you just did. Thanks for doing the work that you do.

Oh, thanks for having me on. I really appreciate it. It's been fun.

All right, folks, I'm Ben Greenfield along with Emily Kaplan from brokenscience.org signing out. Have an incredible week.

To discover even more tips, tricks, hacks, and content to become the most complete boundless version of you, visit bengreenfieldlife.com. In compliance with the FTC guidelines, please assume the following about links and posts on this site. Most of the links going to products are often affiliate links, of which I receive a small commission from sales of certain items, but the price is the same for you. And sometimes I even get to share a unique and somewhat significant discount with you. In some cases, I might also be an investor in a company I mention. I'm the founder, for example, of Kion LLC, the makers of Kion branded supplements and products, which I talk about quite a bit. Regardless of the relationship, if I post or talk about an affiliate link to a product, it is indeed something I personally use, support, and with full authenticity and transparency recommend in good conscience. I personally vet each and every product that I talk about. My first priority is providing valuable information and resources to you that help you to positively optimize your mind, body, and spirit. And I'll only ever link to products or resources, affiliate or otherwise, that fit within this purpose. So, there's your fancy legal disclaimer.