Why We Wanted to Retract Our Own Paper

The data was coming back and it was not looking like the dataset that was provided to us.

The initial Clearly dataset is overturned.

That was the big, that was the big mind-blowing moment.

If they were confident in the quality of their analysis, why would they just not do a check?

In the top 10% of the top 1%.

No good deed goes unpunished.

Wait, what?

The analyses outside of theirs disagree with their analysis. So I want to be clear, what I'm gambling here isn't some like social media points. It's my heart too. All right everyone, this is a very special conversation, really one that's going to be cathartic for me and my friend, allow us to put a lot out on the table. It's been one I've been wanting to have for 11 months now, almost a year, since we published a very controversial study in Jack Advances, um, which to the surprise of some, we are now pushing to withdraw. Retract. Now, when I say we, I mean like myself, my coauthor and colleague here, Dave. This is a study that we are actually proud and enthusiastic to pull from the literature. And this discussion is going to explain in part why. So if you don't know Dave, actually, Dave, why don't you introduce yourself?

Yes. My name is Dave Feldman and I'm an engineer who got obsessed after his cholesterol levels went high on a ketogenic diet. Started doing a bunch of n-of-1 experiments, ultimately became, you know, obsessed with trying to find out the reason why. But also the other important question, which we'll be talking about a lot here, which is the question of risk. You know, do people like you and I who see our LDL level go way high on a ketogenic diet, are we at a higher risk for cardiovascular disease? And that's why we did the Keto-CTA study in the first place.

Yeah, so some of my audience might be laughing because they might either know you really well Um, or, you know, well, they probably just know you really well, to be honest. But to describe you for those who haven't heard me talk about Dave, there's a lot of ways I could describe you. I think from a personal level, like, you are one of, if not my best friend, you feel like a brother to me, genuinely. Um, but also an inspiration. And I don't say that lightly. I'm someone who grew up in academia, meeting a lot of very distinguished scientists and always wanting to see the intellectual integrity behind those esteemed people. And often it was very disappointing. There's a lot of smoke and mirrors in academia. And so when I see people with like true intellectual integrity who just want the freaking answer, that cuts beyond any credential. And the fact that Dave comes from the outside of academic medicine, that he's not a formally trained M.D. or Ph.D., although You've now published more in the lipidology space than several people who have been prominent critics. We'll set that aside for now. You attack it with such like a genuine ferocity of curiosity that I know very few people to match. Maybe myself and our third colleague who isn't here, Adrian, who's just like, it's always just about asking the hard questions and getting the important answers. So this whole area of research really was your brainchild, the studying of these people like me and like you who go low carb and their cholesterol levels go through the roof. Now, this discussion isn't going to be about mechanisms, but more about the story of the study that you got up and running and that I took part in to assess the question, well, what is the risk profile? And for the sake of, of time, I'm just going to kind of like jump to the chase. There was a study that we were both a part of that took 100 people, lean mass hyperresponders who had gone low carb. Their LDL cholesterol had gone through the roof. What was the mean at baseline? Like 272 LDL, like the top 0.1% or something.

It's 2. So the— it depends on which paper we published because, as you know, we're going to probably also talk about the MATCH analysis that was before that, which was a subset.

Yeah.

The majority still. But of the last paper we published, it was 254 was the average LDL, and that is in the top 10% of the top 1% of Inhaines. So that's 1 in 1,000 have an LDL level average at this degree. But yeah, even then, as you know, we also have a widespread, which we'll probably also talk about too. But if you go by the average, the average is 254.

Okay, so very, very, very, very high is the bottom line. And the question is, are these people at increased cardiovascular risk? So the design of the study was, in short, to follow people over the course of a year with high-resolution coronary CT angiography. So not just looking at calcified plaque, but also soft plaque and see, you know, what is the progression and what predicts progression as well. So the paper we published was actually the we've published several, but it was a second in a series. But I want to focus on the April 7th Jack Advances paper where the focus was, as in the title, that ApoB and LDL don't predict plaque progression. Basically, baseline plaque predicts future plaque. So it was like a plaque snowball. But the main novel finding, which people tend to overlook, is that the common culprit, LDL or ApoB, doesn't predict plaque progression. Now, as we'll get to, that's a very robust finding, but— and here enters the controversy— there was measured progression over the course of a year. And would you say it was large based on the initial report?

It, it was larger than what I would have expected. This is— so to restate what you just said, and I think just to make it easy for everyone, In this paper that we published last year, we then had two analyses. The first one that came and the one that we shot the movie around is called the semi-quantitative analysis. And that's, I lovingly call the horseshoes and hand grenades version. It has to do with scoring. It's very quick and easy to do. The second one was this CLEARLY analysis and that's quantitative. Well, when we published this paper, there were three different things worth noting when looking at those two analyses together. The two biggest robust findings are the ones you just mentioned, that plaque predicts future plaque and that LDL and ApoB have no association in either of these two analyses. But I did bring this up, as you know, you were part of those conversations earlier on. I was a little concerned that there was a difference enough between the semi-quantitative and the clearly quantitative that did bother me because we didn't see this often in other studies that had both a semi-quantitative and a quantitative. They usually very closely correlated with each other. So in the quantitative CLEARLY analysis, the majority of people— I want to say it was 55, if memory serves, something like that— were under a PAV of 1.0, and therefore it would be fair to say that the majority were low, right? Or lower than rapid progressor.

So I— sorry to interrupt, but I want to actually double-click on this because just, you might think it's simple to be like, hey, we're just going to measure how much people progress and then say, is it high or low? But that actually becomes a very difficult calculus because there's a lot of different things you can look at. And for context on background, uh, my entry in the project wasn't until well after study design, and what became the point of emphasis in the first paper we published were markers that have better established cut points for what's a high and low responder. So one, you just mentioned PAV, percent atheroma volume, and based on that metric, most people were not rapid responders. So it would have been accurate or remains accurate even based on that dataset to say most are not rapid responders or rapid progressors. Um, but it becomes complicated then when you look at other metrics like non-calcified plaque volume, the absolute change, and then the percent change. And we don't need to get into all the differences here. Bottom line is different metrics tell— go ahead.

But let's hold on the rapid progressors for one second because it would be fair, I think, to point out that, wait, in the semi-quantitative data, the one I just mentioned, there weren't that many rapid progressors. Based on total plaque score, which is what it used.

Very few, right? Yeah.

I want to say it was 6 or 7 at something. Yes. Out of 100.

Right. And experts going through and saying, hey, at these points in the vessels, I'm giving a 0, 1, 2, 3 score.

Right.

And I think it sums up to 15. And I don't think that there were— there was a single person with like double-digit advancement.

The total possible score is 45. It's a 0 to 3 in 15 points. So in, in short, how many people's scores changed from the baseline to the follow-up for, I want to say, a 3 or higher. And I want to say, and if memory serves, it's like 6 or 7 total, which would be basically expected in any middle-aged population.

Middle-aged population, you're probably going to see out of 45, and I don't think there were any had more than like, like, no, there were no single digit additions. So more than single digit additions. There's no, there was no plus 10.

Oh, definitely.

Right. So, so bottom line is what we're saying. I'm sorry. I'm getting us lost in the weeds here, but there's the semi quantitative where an expert goes through and reads it. And then there's this AI read that was done by Clearly. It wasn't really the focus of the initial manuscript. Because the initial manuscript was plaque doesn't— plaque predicts plaque, Appleby doesn't. And there were always going to be subsequent manuscripts, including other analyses by Q-ANGIO, which is the pre-specified methodology by which we were going to have precise quantification of plaque progression, right? Yes.

I mean, to be fair, all of the ones after the semi-quantitative are technically quantitative analyses, but Q-ANGIO, which we haven't talked about yet in the chronology, but it's what comes after the April 7th paper, is the one that takes the longest. It's more labor-intensive. It's considered a quantitative analysis, but importantly, it's the one that is literally in our methods paper. It is the pre-specified methodology. So the pre-specified methodology being Q-ANGO was very relevant because until that arrived, we weren't, we weren't technically done with the scope of the original study. So We're publishing papers as the data comes in. This one was relevant though, and I, I do want to make sure this doesn't get lost, in that the two things for which the datasets agreed, all of us including me agreed, were absolutely very novel findings, and we were excited to put that part out. I did have some reservations with regard to the third thing, with regard to that proportional difference like I just talked about, for example, with rapid progressors in, say, the semi-quantitative versus the clearly, because I would have thought that they would match more. So that said, I was looking forward to the point where we would have more analyses to better determine which of these two makes more sense.

So I want to add some resolution to what you just said. The first thing is, or what you alluded to, there's a series here. That April 7th paper wasn't meant to be the be-all end-all. It was like, hey, we have this cool finding and we want to focus on this narrative for this first paper, which was that plaque—

yes.

Yeah. The— that ApoB and LDL don't predict plaque progression despite the study having— you said this, and I'm confident you're correct— the largest spread in LDL of any prospective imaging study ever, period.

That I've been able to find.

Best response. So you'd expect to see an association between LDL or ApoB and plaque progression, and there was nothing. And just to double-click on this— spoiler for— there were then multiple different analyses. That has been the most consistent finding across every analysis, that LDL and ApoB do not predict plaque progression in this population, which is hugely novel and was the purpose of the original study. Now, the other thing that you pointed out was that you had questions and concerns about discrepancies in the data. What people— I wouldn't expect them to fully appreciate is you'd think everybody that's going to be an author on the paper, including one of the senior authors, you, and one of the first authors, me, would have all the information. But that was not the case. And the reason that wasn't the case is because we were affiliated with the study through the funding body, your charity, the Citizen Science Foundation. And so you very rightly put blocks in place to protect the integrity of the project so it could never be accused that the funder is putting their thumb on the scale. So there were data that you and I did not have before the study was published. And even just for optics, you didn't want to push too— don't let me speak for you, tell me if I'm wrong here— but you didn't want to push too hard on what you saw as discrepancies because you didn't want to be like influencing a narrative prematurely.

Yeah, it's very, very easy to comprehend what the nature of this issue is if you take us out of it, for both supporters and detractors alike, and just talk about it in the abstract, right? A funder who is an engineer helps to spearhead a study, crowdfunds it, helps work out the design, but is not himself a researcher. Appropriately, he puts together a team that may help determine the protocol, how it's worked, but it's the research center that actually conducts the study. They get the participants, they scan them, they collect the data. They then let the funder know what the analysis is. And at that point, the funder can be part of authoring the paper, but only to a degree in which it's appropriate. You don't, for example, want a funder who can then say, I don't like those results. They don't make sense to me. They please keep reworking them until they make sense to me, right? I would be critical of somebody who did that, who after they got their data— so it's absolutely been the understanding between the Citizen Science Foundation and Lundquist Institute that as these data come in, we appropriately are going to publish them. But also importantly, I was only ever going to get access to the raw anonymized data at the point that it was published. That's the contractual agreement between the Citizen Science Foundation and the Lundquist Institute, and that's the most appropriate way to do it. There shouldn't be— funders shouldn't have access to the raw data. Go ahead.

I'm smiling because this is the best example I can think of, of no good deed goes unpunished, because I want to clarify, there aren't hard and fast rules here. So I was reading a paper recently about what is predicted to be a trillion-dollar drug.

This is going to be a trillion-dollar drug.

Trillion dollar retatrutide, this new generation GLP receptor agonist. It's a GLP, GIP, and glucagon agonist. And there was this misinformation that's been going around the internet about it being not only a gangbuster weight loss drug, but that it preserves lean mass. And you'll see where I'm going in a minute here. I tried to source the origin of where this myth came from, and it's a line in the abstract of the paper that reports on body composition about reassurance about no more lean mass loss taking this medication despite more weight loss. Now, I remember reading that line, I'm like, and seeing the data, I'm like, if I hired a team of lawyers to write something to plant a seed of misinformation, but that was technically correct and legally sound, it would be this. Now, this is where I'm going. When I then look in the paper It says that Eli Lilly, the drug company that not only funded the trial but was involved in data analysis, data interpretation, and writing the manuscript. So like at every stage in the process, this drug company was involved in the process, had their thumb on the scale. So what I'm saying here is it should be standard practice to create these integrity barriers. And Dave, being an actually good scientist, went out of his way to do it. However, the catch is then you are entrusting the data to another party and can't do certain validation checks until after the fact. And that is exactly what happened. So Dave had some curiosities prior to April 7th. The paper gets published, then you get the hands, your hands on the data. What happens next?

Let's actually take a step back 'cause one of my proudest moments from last year is a humbling one. We got tested because the critics were right about something that I appreciate that we embraced, acknowledged, and immediately put a correction towards. So I genuinely thought, I genuinely thought the percent change in non-calcified plaque volume, which none of us were very excited about given what we not only learned but also where the, where the whole field of science was going—

because somebody's going to click clip that out— not excited about was not— we we saw it and we were disappointed and we want to hide it. It was the metric itself was like it had problems as a metric given that people were all below baseline plaque. So it makes the denominator very low, skewing the actual number. So it didn't— and there also aren't good thresholds for what is rapid progression, which is why the emphasis changed to other markers like PAV. But bottom line is there was— go ahead. Sorry.

No, and this is, this is to, to make, this is to encompass the larger point. If you look right now at an imaging study, right this moment, that's been released in 2024, 2025, 2026, it's going to be focusing a lot on PAV because percent atheroma volume is where the whole field of science went. Percent change in non-calcified plaque volume was a bit of an artifact of the original design But I appreciated that a number of critics said, hey, you actually should, given that even if it was, you know, done from like a different team, you and, uh, Adrian weren't on board then, you would have appropriately pushed back on that.

You realize, because I honestly didn't know it was the primary outcome. That might be embarrassing for me to admit because probably I should have gone back and be like, what is the exact to the line? Is it absolute? Is it percent? But like, It didn't even occur to me that would have been the listed primary outcome. Anyway, right, go ahead.

And so to that extent, I owned the fact that I'm the one through line between the different teams. I myself should have also caught that, that was the— that we should have had the numeric value of it, even if we don't think very highly of it like the rest of the field now. And I actually did think it was in Table 3, but, but critics were correct when they dug— they were trying to find anything about our paper— in finding that, that, that was worth calling out. And so we immediately got tested on that, had our correction set up, and we're ready to fire it over for the, uh, for the journal, because this is something that does happen. But I didn't want to just have the correction and then be like, hey, corrections posted in a tweet. I went so far as to actually have interviews where I was facing tough questions and so forth, because I felt like it was worthy of highlighting to that extent, right? So that, that's an important piece, because to me, I think that's part of the resume of a scientist. A scientist, a good scientist, will not only make mistakes, but a good scientist will acknowledge when mistakes have been made, own them, right? Because they cannot call out other people's mistakes if they themselves can't own when they make mistakes, right?

Yeah. I mean, to me, like, when it happened, it was almost— it was like a nothing burger. It's like, oh, it's in the figure. Oh yeah, I guess a number was left out. Here are all the data. And then, like, released a huge tranche of data publicly. So that should have been a nothing burger, but it became a hangup and then the seed of what spiraled into, uh, a big hit campaign. I don't know what to say, but I want to actually get on because This is like where I wanted to start the interview, talking about what happened after the paper.

Well, so we're— so we have all this activity happening between April 7th and a very important date to me, one I'll never forget, which is April 18th. On April 18th is when the Lundquist finally delivered the anonymized dataset to the Citizen Science Foundation. So I, as president of the Citizen Science Foundation, again in the contract I get the data in hand. I finally have the raw anonymized data, and I want to say this with full acknowledgment that I understand very smart people in statistics and imaging who were working on this before probably just got very distracted by what they were looking for. But me, I was already in full sleuth mode. I found a number of issues with the dataset, a number of issues to where I basically compiled in 24 hours a report that has 24 of those issues. Now, the two biggest ones I highlight a lot for laypeople because that one's a lot easier to understand, hopefully. But even at that time when I was highlighting it, I was having some difficulty getting it across to other people internally. But the most important thing is that once I recognized these issues, I brought them immediately to the attention of the Lundquist Institute and to CLEARLY themselves, the ones who provided us this data set. That I was having issues with. Now, after I got the dataset, wrote the report, etc., we were able to secure a meeting that you and Adrian were able to attend with the CMO at the time of Clearly, uh, may still be CMO of Clearly, who's also an author on our paper, who's on an author on this April 7th paper.

I did not know because of interest was not disclosed, and that is not— that is I mean, I, I raised that because it's relevant. It's relevant to the story as it unfolds. Like, just for Terence Parise's sake, and I'm not suggesting any malintent here, but if you're the chief medical officer of a company and your company's technology is being used for the study, like, that should be disclosed in the paper. The fact that other members of the author team weren't aware of that relationship, I just think is as generously as I can put it, a mistake that should have been rectified and owned.

In the moment, was likewise surprised, but at the same time, you could understand why Nick— I would actually almost think this would be to our benefit at the moment. Like, oh, okay, then this, this should be able to be quickly cleared up. That— no pun intended, right? Where, okay, well then if these anomalies have something to do with something that happened in the pipeline because of how different this dataset is from all other imaging datasets I've ever been able to find, right? It's— let me go ahead and pause to, to make the two main things that everyone should understand, right? One, all, all imaging, longitudinal imaging datasets in every other study, even in the placebo arm, have some number of regressors. That is to say, not all of them only progress. Some, a subset, will have less plaque in the follow-up scan than on the first. Now, some of that's attributed, attributed to true regression, such as these very aggressive, uh, statin trials. They would say it's attributable to true regression. However, some of that is attributable to noise. If you have low levels of plaque at baseline and there's just a small change, it's very difficult for a— for the, the algorithm to make out which has more plaque because of how close they are to each other. So you expect noise, you expect a wobble, you expect a bi-directional scatter. Some is up, some is down. That's normal. That's just what happens with device error. Right? But CLEARLY dataset, in its final confirmation, had no regressors. Not one person had less plaque in the second scan than the first. That concerned me because then I, I started becoming concerned as to whether or not it might be fully blinded. How could it so perfectly only have positive results and no negative results, right?

Well, let me, let me put my mind, my, my critic hat on and say, well, this is a group with unprecedented high LDL, so maybe this is just a demonstration of how high risk they are. What would you say to that?

And I had these conversations internally. You were part of some of them where I said, you can absolutely believe in the lipid hypothesis. You can even believe that all of our people are in fact only progressing, but you still can't get around what should be the noise below the noise floor. If, if I, I like to use, um, a, you know, a bathroom scale as an example. Bathroom scale could be off by, say, a quarter of a pound. So that means if you grab a bag of marbles and there's 1,000 of them in there, and you grab another bag of marbles that has 2,000, okay, you might be able to see the difference between the two. But what if you've got a bag of 1,000 marbles and a bag of 1,001 marbles? If you weigh both bags 100 times, are you going to have a number of times where the 1,001 marbles are going to show less than the 1,000 marbles? Yes, because that's the error you would expect. That's the noise below the noise floor. The other one is actually very well established, which is known as the power of zero. You and I— I think you've even done videos on this, this point, right? If you have a zero calcification in your coronary arteries, as shown by a CAC scan or in a CT angiogram, if there's no calcification present then the likelihood of a future event, or for that matter more plaque progression, is substantially lower than if you do have calcification in your coronary arteries. Only in the CLEARLY dataset that we had had by this point in time is it the reverse, and by a lot. The people who had no calcification had 3.5 times more plaque progression than those who had calcification. Than those who had a positive CFC.

This is the opposite to all, like, the prevailing literature. This pattern is going in the completely wrong direction.

And we're talking a huge magnitude.

Yeah, yeah, so this, I think, of the two anomalies, I mean, there are many more you picked up on, but of the two, I think that's probably most likely to resonate with people. We can get to the other one, um, with respect to the noise floor, but just, there are patterns here is the point that don't fit up. With the existing literature that Dave has identified. Let's imagine there's two possibilities here. There's a massive bias effect in the dataset, and we'll get to why I think, in addition to these anomalies, that's more likely, or they're all high-risk participants. Now, as we mentioned, this wasn't the pre-specified methodology, so I'll just kind of reveal the spoiler people probably already know. But given the anomalies in the dataset, the Citizen Science Foundation funded additional independent analyses that were fully blinded. So Q-ANGIO, the pre-specified methodology, was completed, and then an independent analysis by HeartFlow was completed that was completely blinded. And the punchline is these agreed with each other and disagreed with the progression on the initial CLEARLY dataset. So now we have one dataset which is an outlier for many reasons against the entire literature base and two other datasets that independently were conducted that agree with each other. One was the pre-specified methodology, the other one was completely blinded. They agree with each other and not the initial dataset. That alone should have been enough to call the initial ClearLee analysis into question. The thing that really threw me off when I found out about it, and I really want to emphasize when we found out about it, you and me, because this is not within our scope of awareness, when the paper was published, is that the initial analysis was not properly blinded. Can you explain what that means and why it is so enormously important?

So blinded, you hear about a lot with studies, particularly things like double blind. But in this case, blinded is crucial because who's ever doing the analysis of the 200 scans we give them They should be able to know the scan pairings, that two scans belong to a subject, but they shouldn't be able to know the order of those scans. If they know the order of the scans, that can be a problem, and let me explain why. If you're somebody who already believes in the lipid hypothesis, if you know you're working on a study where there's sky-high levels of LDL and you have scan you have baseline on the left and you have scan follow-up on the right. And let's say that you do, you know, your best version of analysis of both scans, but scan 2, you're coming up with lower numbers on the right with the follow-up. Even if you don't intend to do this, even if it's just unconscious, you might go, maybe I didn't give it a close enough look. And so you look a little bit more at the second scan, you realize, oh, actually this artifact might have been different if I, you know, work the tool a different way or things along those lines, right?

That's assuming a human's reading it to some extent or influencing it. I think Dave's just using that by way of analogy. If you don't, if you have things labeled with dates, so there is a clear pre and post, there is just an opportunity for a skew, whether it's human, whether it's technical.

This may apply to AI. That's where I was going. You can't— don't assume because it's AI. That it will not account for what the chronology is, particularly if it's based on like neural networks and so forth and has to do with training, right? This is again, doesn't matter who's doing the analysis.

AI reading melanoma, where they were trying to train AI to read, um, skin lesions and figure out what was a cancer. And it turned out it was phenomenal. It got 100% right. Like, this is fantastic. You know what happened? It was, there was a ruler. If you had done the biopsy and the, the people that were training it knew it was melanoma, they had left a little ruler on the side. So every melanoma was marked with a ruler. So be it a ruler or be it like pre-post dates, be it human or be it AI, having an unblinded read introduces an enormous source of error or bias, potential bias. And so now you have a dataset. That is already showing anomalies versus all the other literature, that has been unblinded, and disagrees with two independent datasets including the pre-specified methodology and a blinded dataset.

And so this is the key thing to not lose. Literally from that first moment that I gave the report all the way to now, Nick, there has been repeated emails at each stage of this between the Citizen Science Foundation and CLEARLY. Where initially, even before these other analyses came in, I get this report before I know about the unblinded. I go, can we get a fully operationally blinded analysis? Just a quality control check. We will pay for any cost if that's the concern, right? But given the anomalous nature of these, I, I'm sure you would agree this data set is unlike anything that's ever existed in imaging before. Could you at least do a quality control check? Presumably it would be similar, right? Then we find out about the unblinding in June. We also find out, uh, we also fortunately, um, were able to commission HeartFlow as it became more and more obvious that they would not do the quality control check.

We wanted any reason. So this has always baffled me. If they were confident in the quality of their analysis, why would they just not do a check, especially when as described here, which are just the facts, it does not look good for the integrity of their process. Unblinded, there are anomalies. It disagrees with multiple independent analyses. Why wouldn't they check their work? And I can't think of a single good reason, not one. Nor do I think they've come forward with them, or has anybody presented any plausible explanation for why they wouldn't do a blinded quality control pass.

The short answer I have that my lawyers will allow me to say is that they stand by the independent analyses outside of our study as validation for the dataset within, in addition to other things that they've stated.

Wait, what? But the analyses outside of theirs disagree with their analysis. So how— what? I'm just—

I'm repeating— I'm trying to summarize faithfully what it is that they've said to us. And I'll just say that there's additional things that they've said as well. I actually thought that by this point they would have made a public statement as to where and why they would choose to not do this, uh, this blinded reanalysis for a quality control check. I want to state one more thing because it's kind of important. At the point that we realized they would not do a quality control check, that's when we engaged HeartFlow. HeartFlow was not originally part of the succession of analyses we were doing, but we knew if we did only one more and it was Q-Angio, even if it was pre-specified methodology, then it would be specified and it was like 2 versus 1. Yeah, it wasn't going to look— I wanted to have at least one more and I wanted it to be AI guided and HeartFlow. And I didn't actually know this at the time. HeartFlow is bigger than, than Cleerly, and it's actually the most validated AI analysis platform, right? Well, HeartFlow— this is the part I want to emphasize— HeartFlow, from minute one, I said, now, can I confirm, if you, if you provide us data that seems anomalous, would you do a quality control check if that were to happen? Or if, as part of an agreement with Clearly, we say we're willing to do multiples with all analyses, do you have any problem with doing repeat fully blinded analyses. And not one single conversation have they said no. They— it's usually been some version of, of course, of course we want to guarantee the quality of, of our work. That's what we sell. We sell accuracy in, in analyses on these CT angiograms.

So, and if anybody has any doubt that this should be done blinded, go ask your favorite AI, should a prospective coronary scan, study be done blinded, right? It will say— I think Grok said it is non-negotiable, it needs to be blinded. And as a matter of fact, CLEARLY has a record of doing blinded analyses. There was a 2023 paper in another JAC journal, the same umbrella, that also had James Earls actually in this one as senior author, and it says repeatedly that they did a blinded analysis. So I can't think of a single reason why CLEARLY didn't do a properly blinded analysis on this study and why they wouldn't do it when offered compensation, especially when their dataset is called into question by these multiple independent analyses. Now at this point, I wanna say behind the scenes what's going on because we've already revealed it. This is the point where we no longer have confidence and want to withdraw the paper. So I said we want to withdraw, retract, whatever terminology you wanna use. People get a little bit tied up on which is which, but I think they're basically synonymous. We want this paper pulled and this dataset pulled because the CLEARLY analysis is not reliable based on information that came to light to us after the January 7th public— April 7th publication. So the— I should also mention the retraction was approved in— dehiva.

Oh, in January 12th. So let me fill in some lines here because we've been talking in the timeline up until it was June in which the heart flow analysis was completed. It was August in which the Q-Angio analysis was completed. And I, I want to say this in your defense, it really— in everyone's defense, as you well know, behind the scenes, I'm saying because it's just true, listen, the whole world and history is watching us. As strongly as we feel now about this CLEARLY analysis, we know the level of attachment detractors have— and this includes even very intellectually honest detractors to the CLEARLY analysis from their perspective, they feel that it's proof positive that, you know, lean mass hyperresponders are high risk. So we didn't just need to get these additional independent analyses. We needed to give even more of this accumulated evidence back to CLEARLY and say, listen, the report I sent you where I identified the problems with the patterns. Now these additional independent analyses of the same 200 scans are filling in the patterns I was saying we should be having. And I kept turning around to you and Adrian, I kept saying, please be patient. Sorry, Citizen Science Foundation is still back and forth thing. And now there's lawyers involved because lawyers did get involved. Can't talk too much about that at this time. But only, only after we took their no as a no, gave them every opportunity we possibly could over months and months, did we finally say, okay, I guess at this point we are going to have to make the, um, we're going to have to make the choice to withdraw. And we wanted to see if we could withdraw while also presenting the manuscript, which I'm happy to say is the manuscript that includes HeartFlow and Q-Angio, and it's right now up in its, in its, in its preprint for what it's worth.

Preprint's available, people can read it. Um, but you know, the original dataset with the CLEARLY analysis was not reliable. The withdrawal was approved back in January. At the time we're recording this, it's March 10th. It still hasn't gone live for reasons I personally do not understand. There's a little bit more information behind the scenes that I won't get into. I might get in legal trouble for that, but that aside.

Then there's a plot twist.

There's a plot twist. I want to say one thing though, before we get to that plot twist, which is the, the, the retraction has been approved. We put in the request for withdrawal. We've wanted it for a long, long time. Um, and I really don't think that there's any debate about the reason for the withdrawal, which I have said multiple times in emails to the journal being like, we agree here. There's no debate. The reason we're withdrawing is the Clearly Analysis that specifically is not reliable. Do you think that there's any quibbling on that? Like, that is the source error.

I, up, up to literally this minute, and since the QAnon data has dropped, I've not heard anybody on any side jump to the defense of the Clearly dataset anymore. It's the, the— and this is before the plot twist, right?

I wanna This is, but this is before, so this is one of the things, this is one of the things is like, there was, I, you know, the phenomenon of the learned helplessness where you like beat a dog enough and it just stops trying to like avoid the beating or avoid the electric shock or whatever painful stimulus you're feeding it. I was at that point where I'm like, wow, all the data are really pointing to the fact that like the CLEARLY analysis is problematic and yet people are attached to it. You know, the— I'm not gonna speak poorly of the journal, but let's just say we're having trouble behind the scenes, you know. And then the plot twist happened and it completely reinvigorated me because it was such— I'm just gonna say it— like a brilliant chess move that I will be clear, I only knew about the night before the news dropped publicly. Dave, what was the plot twist?

Plot twist is a number of participants, uh, actually took their scan— they requested their study scans, the same scans that all of these analyses are based on, got them into their own hands, took them to their cardiologist. Their cardiologist, who has a valid Clearly account, a cardiologist Clearly account, can submit them on behalf of their patient, being our participants in this case. As they're doing that, conversations start happening with me. I go back to the lawyers. I'm wanting to be sure everything is absolutely above board because now this is outside the study. Um, they emphasize, hey, please, you know, keep this extra, extra confidential. I'm like, no problem. I'm Dave Feldman. I'm good at keeping confident. Nick Norwitz knows this one, right? Um, but as I shared with you privately, and I'll share this for the first time publicly, boy was I dying inside because of course I wanted both you and Adrian to know about this as, as soon as it could be, uh, the case because the data was coming back and it was not looking like, uh, the data set that was provided to us. Importantly, the data looks more like those other analyses, like all the other analyses that have been outside of it, except even more so, even though the sample size is small. So here's— here it is. There were 8 participants. Actually, there's been a total of 19 submissions. I'll say that first. Of those 19 submissions, 8 of them were participants that submitted both their baseline and their follow-up scans to CLEARLY. So those were the 8 that we needed to focus on because they have the longitudinal data. Of those 8, the study version of their data, of their plaque progression, their non-calcified plaque progression, if memory serves, is around like 20 millimeters cubed.

Of the 100 that went through the CLEARLY analysis, these 8, which now have a reanalysis by CLEARLY in a blinded fashion properly, Their original read was a progression of around 20 millimeters cubed, right?

Um, and just to— I want to state this to Steelman on behalf of Cleerly, believe it or not. I think that they might say, well, this was a clinician version of the scan or something to that effect. So there may be, there may be a greater degree because there's a, uh, as a core lab function of doing these scans.

But that said, they're suggesting that the quality of the reads they give back to clinicians is worse than the ones they use for research grade.

I think, but I should double-check this, I think that's quite literally the case, that research versions of scan analysis are of a higher quality, relatively speaking.

That said, that is a fluffy excuse. If that's the excuse, okay, fine.

I'm gonna do my best here.

Continue, you're trying to represent them.

Regardless, the magnitude of difference is unmistakably massive. I mentioned the 28 millimeters cubed, thereabouts. The, the median change goes down to 0.7 millimeters, near flat, right?

Less than 1, basically flat zero, and that's the median.

Right, the mean, the average is regression. Is the average is regression because half of the participants turn out to be regressors in the direct, clearly submissions. That was the, that was the big, that was the big mind-blowing moment was, uh, yes, direct submissions had 4 of the 8 who are regressors, yet the analysis they gave us for the study None. None were true regressors in their analysis.

Let me emphasize this as strongly as I can. So forgive me, I'm gonna get on my soapbox for a minute, but since this information has come out, people have gotten hung up on a lot of things and we can get into some of those things, but one of those things is, oh, it's just 8 participants. First, let me explain the probability of getting 8 regressors from the initial, initial dataset. From 8 submissions pre and post, like the probability in the initial dataset of having 0 regressors, then having 8 basically at random and having 50% of those be regressors. The analogy I've been using, hopefully it sticks, is I genuinely think there's a higher probability that I could put on a blindfold and punt a ping pong ball in a hurricane a football field's length into a Red Cell lookup on my first try. The probability of having 50% regressors from those 8 from chance, from the initial dataset, is zero. And then you put it in the context of everything else we discussed. The initial dataset was unbeknownst to us unblinded, which is completely inappropriate and clearly wouldn't do a reblind and reanalysis. So multiple independent analyses were done, including Hartfleisch, which was blinded, Q*Angio, the pre-specified, and now this new dataset. And those all agree with each other. And not with the initial unblinded dataset. The story is so clear here for anyone trying to pay attention.

At least we can say appropriately that we put— we pulled out every stop to try to get this reanalysis done so that people could have the answer. And I still believe our participants deserve that. They're the ones who put their— literally their hearts on the line for these scans. So while it would have been easy and certainly much more convenient, and we would be talking about something entirely different by this point in the year, if in, in April when I had that report, I like took shots at Clearly publicly, uh, you know, showed what it was, we waited till the analyses came in, etc. No, at the end of the day, the right move was, I believe, the ones we went— and don't get me wrong, I'm not saying we didn't make mistakes along the way, or for that matter, that we're not going to continue making mistakes. That's going to happen. The bigger question is, how much did we care about the integrity of all analyses that are done to our dataset? And to that extent, I'm, I'm just so proud of how far we've come. But on top of that, I appreciate how you and myself and Adrian have talked about what's going to be different Moving forward, we— there's been some powerful lessons learned and certainly things we want to put in place to ensure something like this doesn't happen to us again from another provider of analyses. And to that extent, I'm very excited. I'm, I'm excited to be working with HeartFlow for that matter. They've proven to be a partner of the highest integrity. Um, and so I'm, I'm very excited for that as well.

A couple things I want to close on. One, now I'm going to put the spotlight on myself for a moment. People see our avatars. I hope they think we're like authentic people. They see the people behind the avatars. But bear in mind, there's more for each of us than ego on the line here and reputation. Like, effectively, given I am this phenotype, my total cholesterol runs around 700. My choices every day are gambling with my own life, sometimes against the preponderance of evidence of cardiology. And I'm not clueless on these matters. I did get my medical degree from Harvard, who teaches on the conventional point of view with respect to cardiology. So I have been exposed to the gambit. So I wanna be clear, what I'm gambling here isn't some, like, social media point. It's my heart too. So I have every reason to want to get this right. And I'll add to that, when you've been that patient as I have, and I think a lot of people listening have been, and you're even a modicum of a decent human being, the thing you want above all else is justice for yourself and for people like you, because you know what it's like to sit in that position. So there is no universe in which I, or either of us, or I'll include Adrian as well, obviously. Would take advantage of the scenario for any purpose other than to get the patients what they deserve. That is the purpose of all of this, full stop, end of story. Not really end of story, because my last question for you is, what happens next?

Well, you already know this, we want to get the 5-year scans, which may be 6-year scans, it depends on how soon we can get everything arranged, on the original Keto-CTA study. There's already a number of participants who've outed themselves to me, who I've already I've had this conversation with. I've said, you know, we need to wait until we're all the way through IRB, etc., until we can put that in front of them. Funding we've got locked into place, but our companion study, which is going to be quite a bit larger and requires a lot more money, is what we're about to fundraise against with the conference this weekend with COSI. But by the time this is out, it will have already happened, right? And hopefully, hopefully we're this close to closing that funding. I think we're— I think we are. And once we do, then we can kick that off as well. And that will be a study comparable to Keto-CTA, but a little more relaxed, um, with regard to eligibility criteria. And it'll have a control group. And so I'm, I'm very excited to see what that's going to look like.

Yep. And we have the conference, we have another study, we have 3 new papers in the process of production, some under review. Movie, you know, um, I hope this isn't a spoiler, but there's a line in the movie, "The science isn't subtle," and that's where I want to leave it here. Like, this discussion, I realize, is a little niche, esoteric. It's one I've been wanting to have, not for you, my audience— I'm being selfish here— this is— I was having this for me and Dave because it has been such a slog and a journey over these past 11 months, even before that. To be over this hurdle and to be looking forward is such a joy, and it's going to be a really interesting area of research. And the last thing I'll say— I know I said that multiple times— but, um, behind the scenes, like, the growing interest by people who matter is really tremendous. We don't have to say who, but Dave and I were having a conversation earlier this week with a rather influential international doctor who wants to run an independent study in an independent country on this research. So maybe a silver lining of all the hullabaloo is it's getting attention. And for people who are actually putting a little bit of brainpower into observing what's going on, there is no doubt, like, this is a frontier of medicine that deserves exploration. So those are my final words. Dave, do you have any?

Yeah, I wanted to actually bookend what you said at the beginning that was very kind regarding me, uh, with you. I just cannot imagine where our research would be without us if If I'm the, the match that struck this whole thing, you are the powder keg. And I'm just so delighted that your, your absolute diligence with everything, even well beyond this, has managed to increase your profile enough to help us get the word out, to help us get the research out. Because as I know you know all too well, we can't just count on the network of journals, the network of cardiologists, and so forth. Some of them are very interested, but will that carry the day? No. That's why we need things like this documentary. We need both in play. We need the research, but we also need the social media campaign to carry the research, to carry the good data forward. And to that extent, I'm just so thankful that you and all of your brilliance and all of your hard work have taken you to where you are now, and I can't wait to see where you go next as well.