#188 - AMA #30: How to Read and Understand Scientific Studies

Foreign.

Hey everyone. Welcome to a Sneak Peek, Ask Me Anything or AMA episode of the Drive podcast. I'm your host, Peter Attia. At the end of this short episode, I'll explain how you can access the AMA episodes in full, along with a ton of other membership benefits we've created. Or you can learn more now by going to Peteratti md.com subscribe so without further delay, here's today's sneak peek of the Ask Me Anything episode.

Welcome to another Ask me Anything episode, AMA number 30. I'm once again joined by Bob Kaplan. In today's episode, we discuss all things around studying Studies. If you've listened to this podcast or read any of my weekly emails, you know that I place a large emphasis on being able to sift through the noise and find the signal when it comes to various studies and patients papers that are printed both as the studies themselves and unfortunately, a lot of the times when the media reports on them. So if you follow the news, you know there's no shortage of articles that either contradict each other, seem too good to be true, don't make any sense. It can be hard to understand this. So what do you do? Well, we've, as some of you may know, written a series on this called Studying Studies. But we've also tried to tackle some of the bigger things in this ama and we combed through a lot of the questions that many of you have been asking over the previous months and years that relate to this topic. And I think we have enough of them here that we were able to put together a solid episode. This episode's a bit longer than normal. We contemplated breaking making it into two because it's so long, but I think our audience can handle this. And so at least now you have in one place the all singing, all dancing discussion of how to study studies. So in this discussion we talk about a bunch of things. What are they? What is the process for a study to go from an idea to a design to execution? What are the different types of studies out there and what do they mean? What are the strengths and limitations of each of them? How do clinical trials work specifically for drugs? What are the common pitfalls of observational studies that you should be looking for? What questions should you be asking about a study to figure out how rigorous it was? What does it really mean when a study is statistically significant? Why do some studies never get published? What is my process for reading scientific papers? So if you're a subscriber and you want to watch the video of this podcast, you can See it on the Show Notes page. I think that's valuable for this episode because I do refer to some tables, though this is still certainly amenable to audio only. If you're not a subscriber, you can watch the sneak peek of this on our YouTube page. So without further delay, I hope you enjoy AMA number 30. Hey Bob, how are you, man? Looking pretty studious there in the library today.

Hey, Peter, thanks very much. Yeah, just getting some reading in before the, before the podcast.

This is going to be a pretty good one because as you may recall about, I don't know, four or five months ago, maybe longer, I was on a podcast with Tim Ferriss and I don't know how it came up, but I do remember somehow it came up that we had spent a lot of time writing this series studying studies and God, that's been four years ago, I think. But we didn't really have something more digestible for folks on how to make sense of the ever changing landscape of scientific literature and how to kind of distinguish between the signal and the noise of the research news cycle. And I remember after that Tim and I went out for dinner and he kept pressing me on, well, what can I do to get better at this process? Are there newsletters I should be subscribing to and things like that? And while I'm sure that there are, I didn't know what they were off the top of my head. And so I think what we've done here, when I say we, I mean you. What you have done here is aggregate all the questions that have come in over the past year basically that pertain to understanding the structure of science. I looked through the questions last week and I was pretty excited. I think it's going to be a sweet discussion and I hope this serves as an amazing primer for people to really understand the process of scientific experiments and everything from how studies are published and obviously what some of the limitations are. So anything else you want to add to that, Bob, before we jump in?

I agree. I think it's a fun topic. We get so many of these questions that we end up, or at least I do, or to the website where we'll point readers to one of the parts of the studying studies. But I think sometimes just talking about it and explaining it can help a lot. So I think this will be really useful as far as like a question and answer session rather than just reading a blog.

I don't think this displaces that other stuff. I think we go into probably more detail on some things There. But I also think we're going to cover things here that aren't covered there. So depending on how you like to get your info, this could be fun. So where do you want to start?

We have, again, a lot of questions, but I think this question gets to the core of I think what we're trying to do here, which is how can a user or a person who has no scientific background better understand studies that they read in the news or in the publications to know if the findings are solid or not? Especially in today's age, where you can easily see two studies that contradict each other. Coffee's good. Coffee's bad. Eggs are good. Eggs are bad. So I thought we could run through a bunch of questions with the first one that we got here is what is the process for a study to go from an idea to design and execution?

This is a great question. In theory, it should start with a hypothesis. Good science is generally hypothesis driven. I think the cleanest way to think about that is to take the position that there is no relationship between two phenomena. We would call this sort of a null hypothesis. So my hypothesis might be that drinking coffee makes your eyes turn darker. So I would have to state that hypothesis, and then I would have to frame it in a way that says my null hypothesis is that when you drink coffee, your eyes do not change in color in any way, shape, or form. And that would imply that the alternative hypothesis is that when you drink coffee, your eyes do change color. You can already see, by the way, that there's nuance to this, because am I specifying what color it changes to? Does it get darker? Does it get lighter? Does it change to blue, green? Does it just get the darker shade of whatever it is? But let's put that aside for a moment and just say that you will have this null hypothesis and you will have this alternative hypothesis. And to be able to formulate that cleanly is sort of the first step here. The second thing, of course, is to conduct an experimental design. How are you going to test that hypothesis? As we're going to talk about. A really, really elegant way to test this is using a randomized controlled experiment. If it's possible to blind it, we'll talk about what that means. You'll have to decide, well, how long should we make people drink coffee? How frequently should they drink coffee? How are we going to measure eye color? These are the questions that come down to experimental design. You then have to determine a very important variable, which is how many subjects will you have? And of course, that will depend on a number of things, including how many arms you will have in the study. But it comes down to doing something that's called a power analysis. And this is so important that we're going to spend some time talking about it today, although I won't talk about it right now. If this study involves human subjects or animal subjects, you will have to get something called an institutional review board to approve the ethics of the study. So you'll have to get that IRB approval. You'll have to determine what your primary and secondary outcomes are, get the protocol approved, develop a plan for statistics, and then pre register the study. All of these things happen before you do the study. And of course, in parallel to this, you have to have funding. So those are kind of the steps that go into doing an experimental study. And what we're going to talk about, I think in a minute is that there are some studies that are not experimental where some of these steps are obviously skipped.

Yeah. One of the questions we got was what are the different types of studies out there and what do they mean, for example, observational study versus a randomized controlled study. What are the different types of studies?

I think broadly speaking, you can break studies into three categories. One would be observational studies. We'll bifurcate those or trifurcate those in a minute. Then you can have experimental studies, and then you can have basically summations of and or reviews of and or analyses of studies of any type. Let's kind of start at the bottom of that pyramid. I think you actually have a figure that I don't like very much, but.

I was going to say that was one of your favorites.

Yeah, I can't stand it. I'll tell you what I like about the figure. I like the color schema because my boys are so obsessed with rainbows that if I show them this figure, they're going to be really happy. So let's pull up said rainbow figure.

Okay, got it.

Okay, so you can see these buckets here. And again, at the level of talking about them, I think this makes sense. What I don't agree with the pyramid for Bob is that it puts a hierarchy in place that suggests a meta analysis is better than a randomized control trial, which is not necessarily true, but let's just kind of go through what each of these things mean. So looking at the observational studies, an individual case report is first or second paper I ever wrote in my life when I was in medical school was an individual case report. There was a patient who had come into clinic when I was at the nih. This was a patient with metastatic melanoma and their calcium was sky high, dangerously high, in fact. And obviously our first assumption was that this patient had metastatic disease to their bone and that they were lysing bone and calcium was leaching into their bloodstream. Turned out that wasn't the case at all. It turned out they had something that had not been previously reported in patients with melanoma, which was they had developed this parathyroid hormone related, like hormone in response to their melanoma. This is a hormone that exists normally, but it doesn't exist in this format. And so they're cancer was causing them to have more of this hormone that was causing them to raise their calcium level. It was interesting because it had never been reported before in the literature. And so I wrote this up. This was an individual case report. Is there any value in that? Sure, there's some value in that. The next time a patient with melanoma shows up to clinic and their calcium is sky high and someone goes to the literature to search for it, they'll see that report and it will hopefully save them time in getting to the diagnosis.

Your mentor and friend, Steve Rosenberg, I think of him when I think of individual case reports. I think if you listen to the podcast, he talks about this. But a lot of what motivated him early on, I think, were just a couple of cases. I think it gets back to that first question too, about the process for a study to go to an idea to design, to execution, which is to have a hypothesis. You need to make an observation. And so you make an observation, you say, hmm, that's strange. And I think that that's what individual case reports can represent sometimes. This is an interesting observation. It's hypothesis generating for the most part. But it really might kickstart a larger trial or it might kickstart a career. You never know.

Exactly. Now, of course, it's not going to be generalizable. I can't make any statement about the frequency of this in the broader subset of patients. And obviously I can't make any comment about any intervention that may or may not change the outcome of this. So that gets us to kind of our next thing, which is like a case series or set of studies. So here you're basically doing the same thing, but in plural. Effectively, you wouldn't just look at one patient. You would say, well, I've now been looking back at my clinical practice and I've had 27 patients over the last 40 years that have demonstrated this very unusual finding. And Another example of this, going back to the Steve Rosenberg case would be one could write a paper that looks at all spontaneous regressions of cancer. Obviously, spontaneous regressions of cancer are incredibly rare, but there are certainly enough of them that one could write a case series. So now let's consider cohort studies. So cohort studies are larger studies and they can be retrospective or they can be prospective. So I'll give you an example of both. So a retrospective observational cohort study would be, let's go back and look at all the people who have used saunas for the last 10 years and look at how they're doing today relative to people who didn't use saunas over the last 10 years. So it's retrospective, we're looking backwards, it's observational. We're not doing anything right. We're not telling these people to do this or telling those people to do that. And the hope when you do this is that you're going to see some sort of pattern. Undoubtedly you will see a pattern. Of course, the question is, will you be able to establish causality in that pattern? Cohort studies can just as easily, although more time consumingly, be prospective. So you could say, I want to follow people over the next five years, 10 years who use saunas and compare them to a similar number of people who don't. And now, in a forward looking fashion, we're going to be examining the other behaviors of these people and ultimately what their outcomes are. Do they have different rates of death? Heart disease, cancer, Alzheimer's disease, other metrics of health that we might be interested in? Again, we're not intervening, there's not an experiment per se, we're just observing. But now we're doing it as we march forward through time. So this brings us to the kind of the next layer of this pyramid, which are the experimental studies. Divide these into randomized versus non randomized. And of course this idea of randomization is going to be a very important one as we go through this. So a non randomized trial sometimes gets referred to as an open label trial where you take two groups of people and you give one of them a treatment and you give the other one either a placebo or a different treatment, but you don't randomize them, there's a reason that they're in that group. So you might say, we want to study the effect of a certain antibiotic on a person that comes in the er and we're going to take all the people that come in who look a certain way, maybe they have a fever of a certain level or a white blood cell count of a certain level, we're gonna give them the antibiotic. And the people who come in, but they don't have those exact signs or symptoms, we're going to not give an antibiotic to and we're gonna follow them. That's kind of a lame example. You could do the same sort of thing with surgical interventions. We're gonna try to ask the question, is surgery better than antibiotics for appendicitis or suspected appendicitis or. But we don't randomize the people to the choice. There's some other factor that is going to determine whether or not we do that. As you can see, that's going to have a lot of limitations because presumably there's a reason you're making that decision. And that reason will undoubtedly introduce bias. So of course, the gold standard that we always talk about is a randomized control trial where whatever question you want to study, you study it, but you attempt to take all bias out of it by randomly assigning people into the treatment groups, the two or more treatment groups. We'll talk about things like blinding later because you can obviously get into more and more rigor when you do this. But before we leave the kind of experimental side, anything you want to add to that, Bob?

I would add. So non randomized controlled trials, maybe another example, illustrative example, I think with non randomized controlled trials might be you have patients maybe making a decision beforehand, which we'll get into selection bias. But they might want to go on a statin, let's say, and then you give them a choice. And the other ones might want to go on some other drug like Ezetimibe. They're basically selecting themselves into two groups. But you could compare those two groups and see how they do, but it hasn't been randomized. There's a lot of bias that can go into that. There could be a lot of reasons why one group is selecting a particular treatment over the other. And so that's why I think when we get to randomized trials, that shows the power of randomization.

Yeah, exactly. We don't need to go back to the figure. But people might recall that at the top of that pyramid was systemic reviews and meta analyses. Let's just talk about meta analyses, since they are probably the most powerful. So this is a statistical technique where you can combine data from multiple studies that are attempting to look at the same question, basically. So each study gets a relative weighting and the weighting of a study is sort of a function of its precision, it depends a little bit on sample size. Other events in the study, larger studies which have smaller standard errors are given more weight than smaller studies with larger standard errors. For example, you'll know you're looking at a meta analysis. We should have had a figure for this, But I'll describe it the best I can. They usually have a figure somewhere in there that will show across rows all of the studies. So let's say there's 10 studies included in the meta analysis, and then they'll have the hazard ratios for each of the studies. So they'll represent them usually as little triangles. The triangle will represent the 95% confidence interval of what the hazard ratio is, which we'll talk about a hazard ratio, but it's basically a marker of the risk. And you'll see all 10 studies, and then they'll show you the final summation of them at the bottom, which, of course, you wouldn't be able to deduce looking at the figure, but it takes into account that mathematical weighting. So on the surface, meta analyses seem really, really great. Because if one trial, one randomized trial is good, 10 must be better. I know I've said this before, probably three or four times over the past few years on the podcast, but as James Yang, one of the smartest people I ever met when I was both a student and fellow at nci, once said during a journal club about a meta analysis that was being presented, he said something to the effect of a thousand sows ears makes not a pearl necklace. And that's just an eloquent way to say that. Garbage in, garbage out. So if you do a meta analysis of a bunch of garbage studies, you get a garbage meta analysis. It can't clean garbage. It simply can aggregate it. So a meta analysis of great randomized control trials will produce a great meta analysis.

They try to control for garbage, the researchers and the investigators. But I think to your point, with the pearl necklace, imagine if you had, say, 10 trials and nine of them are garbage. One of them is really good, really rigorous randomized controlled trial, and you're looking at the top of the pyramid and you're saying, well, meta analysis is the best. We should be looking at this meta analysis. Meanwhile, you've got that one randomized controlled trial that actually is worth its salt. It's rigorous, et cetera. That I would say, if you had the option, I think you probably would rely more on that one randomized controlled trial, which is lower on the pyramid. So I think that's probably. I think you've told me one of your hang ups with the pyramid because it's not necessarily top of the pyramid. It's going to be some meta analysis of randomized controlled trials.

That's right, yeah. I don't want to suggest meta analyses are not great. What I want to suggest is you can't just take a meta analysis as gospel without actually looking at each study. You don't get a pass at examining each of the constitutive studies within a meta analysis is really the point I think we want to make here.

There's one thing in here that isn't represented, but we had a few questions about it. I think a couple people were asking about what's the difference between a phase three and a phase two or a phase one clinical trial. Do you know what's going on there?

Yes. So here we're talking about human clinical trials. This phraseology is used by the FDA here in the United States. And typically the world does tend to follow in lockstep, but not always with kind of the FDA's process. So if you go way, way, way back, you have an interesting idea, you have a drug that you think is or a molecule that you think will have some benefit. Think of it as a cancer therapeutic. You've done some interesting experiments in animals, maybe started with some mice and you went up to some rats and maybe even you've done something in primates and now you're really committed to this as the success of this and the safety of this in animals looks good. So it's both safe and efficacious in animals. And you now decide you want to foray into the human space. Well, the first thing you have to do is file for something called an ind, an investigational new drug application. So after you do all of this preclinical work, you have to file this IND with the fda and that basically sets your intention of testing this as a drug in humans. And the first phase of that, which is called phase one, is geared specifically to dose escalate this drug from a very, very low level to determine what the toxicity is across a range of doses that will hopefully have efficacy. These are typically very small studies, usually less than 100 people. They're typically done in cohorts. So you might say, well, the first 12 people are going to be at 0.1 milligrams per kilogram and assuming we see no adverse effects there, we'll go up to 0.15 milligrams per kilogram for the next 12 people. And if we have no issues there we'll escalate it to 0.25. Da da da da da da. You'll notice, Bob, I said nothing in there about does the drug work? These are going to be patients with cancer. If this is a drug that's being sought as a treatment for colon cancer, these are gonna be patients that all have colon cancer. They're often going to be patients who have metastatic colon cancer. So these are gonna be patients who have progressed through all other standard treatments and who are basically saying, look, sign me up for this clinical trial. I realize that this first phase is not going to be necessarily giving me a high enough dose that I could experience a benefit and that you're really only looking to make sure that this drug doesn't hurt me. But nevertheless, I want to participate in this trial. If the drug gets through phase one safely, then it goes to phase two. And the goal of phase two is to continue to evaluate for safety, but also to start to look for efficacy. But this is done in an open label fashion. What that means is they're not randomizing patients to one drug versus the other. Typically they can, but usually it's now we think we know one or two doses that are gonna produce efficacy. They were deemed safe in the phase one. We're now going to take patients and give them this drug and look for an effect. And a lot of times if there's no control arm in the study, you're going to compare to the natural history. So let's assume that we know that patients with metastatic colon cancer have on standard of care, have immediate survival of X months. Well, we're going to give these patients this drug and see if that extends it anymore. And of course you could do this with a control arm, but now it adds the number of patients to the study. So again, typically very small studies can be in the 20, 30, 40, 50 range, maybe up to a few hundred people.

And that one, Peter, I think is probably a good example of if you have the non randomization, this might be a case where say it's an immunotherapy and people know about the immunotherapy and it's been really effective, gets approved for a particular cancer, let's say. And there are a lot of people that know about it, and there are cancer patients that know about it and they want to get that treatment, but it's not approved. They're talking to their doctor, maybe they're online. They might enroll in one of these trials because they really want to try the drug and maybe they, they might Believe in it more than some other treatment.

Yep. There are lots of things that can introduce bias to a phase two. If it does not have randomization, again, the goal would be to still randomize in phase two because you really do want to tease out efficacy. So if a compound succeeds in phase two, which means it continues to show no significant adverse safety effects, which by the way, doesn't mean it doesn't have side effects. Every treatment has side effects. It's just that it doesn't have side effects that are deemed unacceptable for the risk profile of the patient. And it shows efficacy. So really you have to have these two things. You then proceed to phase three. Here, a phase three is a really rigorous trial. This is a huge step up. It's typically a log step up in the number of patients. You're talking potentially thousands of patients here. And this is absolutely a placebo controlled trial, or not necessarily placebo, but it can be standard of care versus standard of care plus this new agent. But it is randomized whenever possible. It is blinded. And with drugs, that's always possible. And these are typically longer studies because you have so much more sample size, you're going to potentially pick up side effects that weren't there in the first place. And of course, now you really have that gold standard for measuring efficacy. And it's on the basis of the phase one, phase two and mostly phase three data, that a drug will get approved or not approved for broad use, which leads to a fourth phase, which is a post marketing study. So phase four studies take place after the drug has been approved and they're used to basically get additional information because once a drug is approved, you now have more people taking it and they may also be using this to look at other indications for the drug. We talked about this recently, right? A phase 4 trial with semaglutide being used to look at obesity versus its original phase 3 trials, which were looking at diabetes. The drug's already been approved. This study isn't being done to ask the question, should semaglutide be on the market? No, it's on the market. It's basically expanding the indication for semaglutide in this case so that insurance companies would actually pay for it for a new indication. But given the size and the number of these studies, you're also looking for, hey, is there another side effect here that we missed in the phase three?

Right. And it might be the particular population that might have a different risk profile, you might have a different threshold.

That's right. Because you're not doing this in patients with type 2 diabetes. You're doing this in patients who explicitly don't have diabetes but have obesity. Different patients. We're going to see something different here. So yeah, so anyway, that's the long and short of phases one, two, three and four.

Okay, so going back to observational studies, are there any things that you look for in particular that will increase or decrease your confidence in it? Whether it's a pearl necklace or garbage.

Thank you for listening to today's Sneak Peek AMA episode of the Drive. If you're interested in hearing the complete version of this AMA ama, you'll want to become a member. We created the Membership program to bring you more in depth exclusive content without relying on paid ads. Membership benefits are many and beyond the complete episodes of the AMA each month. They include the ridiculously comprehensive podcast show notes that detail every topic, paper, person and thing we discuss on each episode of the Drive. Access to our private podcast Feedback the Qualys, which are a super short podcast, typically less than five minutes, released every Tuesday through Friday, which highlight the best questions, topics and tactics discussed on previous episodes of the Drive. This is particularly important for those of you who haven't heard all of the back episodes becomes a great way to go back and filter and decide which ones you want to listen to in detail. Really steep discount codes for products I use and believe in but for which I don't get paid to endorse and benefits that we continue to add over time. If you want to learn more and access these member only benefits, head over to peterattiamd.com subscribe lastly, if you're already a member but you're hearing this, it means you haven't downloaded our member only podcast feed where you can get the full access to the AMA and you don't have to listen to this. You can download that at Peter attiamdash.

You can find me on Twitter, Instagram and Facebook, all with the ID Peter atiamd. You can also leave us a review on Apple Podcasts or whatever podcast player you listen on. This podcast is for general informational purposes only and does not constitute the practice of medicine, nursing or other professional healthcare services, including the giving of medical advice. No doctor patient relationship is formed. The use of this information and the materials linked to this podcast is at the user's own risk. The content on this podcast is not intended to be a substitute for professional medical advice, diagnosis or treatment. Users should not disregard or delay in obtaining medical advice from any medical condition they have, and they should seek the assistance of their healthcare professionals for any such cond conditions. Finally, I take conflicts of interest very seriously. For all of my disclosures and the companies I invest in or advise, please visit Peteratti md.com about where I keep an up to date and active list of such companies. Sa.