Jon Deeks - why are systematic reviews of diagnostic tests needed but so complicated

Delighted to be invited to talk. This title was given to me to talk about why are systematic reviews of diagnostic tests needed but so complicated? As Patrick's highlighted, the Cochrane Collaboration has started publishing these reviews. This was the page from the Cochrane Library. You can pick up diagnostic reviews which have a— That's a good one. Little green box on them. So there are now, as of last issue, 11 reviews and 60 protocols. So they're a little bit of a needle in a haystack alongside the 7,000 other reviews, but they're starting to come out as evidence which some people, maybe NICE, will be looking at to look at the accuracy of tests. Why are they complicated? Well, I thought about this talk. I could give a very boring talk on these topics, that the literature is badly indexed, that there are lots of things which are wrong with primary test studies, that we have to do difficult meta-analyses which requires expert statistical input, and lots of the primary studies are badly reported. That is part of the reason they are complicated. They are harder to do for those reasons. But actually what I wanted to talk about was thinking about how complicated they are to use, because perhaps that is the most important thing for this audience. If you looked at a diagnostic test accuracy review, you may sit there and say, I want to know whether or not I can use this test, but having read this article, I am not sure whether this review tells me that. I just wanted to go through in this session some reasons why we have this problem when you are looking at this, that you might feel that these articles are not informative. This is in contrast to systematic reviews of interventions. I just picked out a random intervention review. Should we floss when we brush our teeth? And if we look at this review, you can see it from the back of the room, it's all about looking at an outcome which is gingivitis. You probably can't see that at the back of the room. But you can tell this diamond's to the left of the line. It actually tells us that favors flossing. So without actually looking at all the detail, you can get the bottom line message here that the evidence is towards saying flossing is a good thing. You might then think a little bit harder about it and say, well, I want to know about the results in absolute terms. That was an index of a score I showed you there as the outcome. What does that actually mean for me? You might ask those questions. You might think about, well, I need to look at the harms. That was just one outcome. Maybe there are harmful outcomes of flossing. You can have a think about that and see whether you can think of one. And then you might think, actually, were they good quality studies? Is there publication bias? Is there heterogeneity? And at the bottom of that, you've got through that list, you should have done your evidence appraisal and you'll think whether or not I should floss or not. And probably the answer is yes from that review. Okay, look at this compared to a diagnostic test accuracy review. This is a topic which I'll use twice in this talk and it was mentioned in this room I think about 6 hours ago in the morning session. It's looking at whether we should use pulse oximetry testing on newborns within about 6 hours of birth, within the first 12 hours of birth, to detect babies who've got congenital heart defects. And there was a review published this— in The Lancet on this, and here are the study results up here on this. This picture doesn't tell you straight away at all whether this is good or bad. There's no neat little text on here saying good this way, bad That way. You have to learn a little bit about what these nasty measures of sensitivity and specificity mean. You maybe should have done that at medical school or something like that, but— It was only taught once probably. And we can add a dot on here which is the average. We compute averages in the same way as we do for intervention reviews and we can put it here. And now we get some numbers, 76.5%. Oh, we're missing a quarter of them. Is that good? Is that bad? We've nearly got everybody. There's only a very low false positive rate. So, there's some information here, but we can't just leave this picture with a very quick message to say this is saying this is good or bad. We can think about the analogous issues. Important to think about the absolute numbers of true positives, false positives, false negatives, and true negatives. This is a very rare condition. So a very high specificity might not be high enough. We have to think about consequences. Patrick told us about that this morning, about what actually does it mean to be a false positive or a false negative in this situation. We want to think about methodological quality. I've put publication bias in brackets and I'll explain a little bit why, why we don't think it's actually such a problem here. That will come up in a few minutes. And we can think about heterogeneity and subgroups. We can do the same sort of analysis and it's a little bit more problematic, but what What I want to talk about are 3 other issues, and they're here on this slide. The first one is actually about the pathways. And we've heard about this already from the talk from NICE, that actually we think a lot about pathways as to where tests sit in pathways, and how they perform changes with that. There's a question about what are we comparing with, and I'll finish by talking about it's not all about accuracy. So when we evaluate a test and when we look at evidence about a test, we have to be clear how and where it will be used. We need to be looking at its applicability of the evidence, and this matters more when we're looking at diagnostic evidence than when we're looking at intervention evidence. When you're looking at a review of patients being treated for a disease, we know at the start they all have the disease. When we start off with these reviews, we don't quite know how people got there quite often. Was it they presented with a symptom? It's a much more heterogeneous group and there's much more ability for the— there to be heterogeneity in what's coming into the study. Let's go back to the pulse ox example. This was an HTA-funded trial. This was mentioned this morning and we ran this trial in Birmingham. We managed to recruit 20,000 babies who all had this pulse oximetry test. We did it in 13 months. Babies don't run away. They're quite easy to recruit to a trial. And of these, we got 24 cases of critical congenital heart disease. And we picked up 3/4 of them, 75%, very similar to the overall figure from the meta-analysis. Is this the right answer? Does this tell us how well pulse oximetry is going to perform? This topic is being debated at, well, not just this minute, but in these current months by the National Screening Committee as to whether this should be introduced. Does this study tell them what they need to know? Well, what we did in this study is we took all newborns who were apparently healthy at birth, didn't go straight to the intensive care unit, They underwent pulse oximetry. If that was positive, they had echocardiography as a reference standard. If it was negative, they followed through the normal clinical examination before discharge route, which is standard care. That would have taken them to echocardiography if that was suspicious, and we ended up with these groups. And we used long-term follow-up to verify whether or not there was no congenital heart disease in these ones who went negative. So there are some babies who are picked up within a year who are— where it wasn't quite clear. Is this the right pathway in which we stick this? Well, actually, there's something missing here. That actually pulse oximetry is the second test which is used. That babies are detected as having suspicious— a possibility of congenital heart disease at the 20-week anomaly or healthy baby scan. So already some patients we had in our study had been diagnosed at having congenital heart disease. We didn't take them out because there were only a handful and spotting them would have been more trouble than taking them out at a later date. And when you analyze the data, there's actually 12 of the 24 were already diagnosed. And when you look at the sensitivity, it's actually quite a bit lower when you've dropped out the ones we already knew about. Okay. Now that's— think about that for a minute. That's going to happen because the ones picked up on the 20-week scan probably have more severe disease than those not picked up. So we have a reduction in the sensitivity. It's an example of making sure that we're actually using the right data and thinking where things sit in pathways. It'll affect here, the sensitivity is reduced from 75 to 58%. Obviously there's a lot of uncertainty around these figures. But it's to make a point that actually it will matter. You put things from the wrong study without thinking about what prior tests have been done or haven't done, you'll get the wrong answers. So that's a bit complicated. You need to look at the studies carefully and understand what's gone on. Second point is compared to what? Now there's a terrible joke which I get told off for telling too many times, but if you ask a statistician, if you'd ask me how's my wife, How's your wife? The answer is, compared to what? We can't say how good a test is, you know, if you answer that question, is pulse ox good enough? Well, it's compared to what? What is the alternative? What is current practice? If it's 75% is worse than we're currently doing, we don't want to use it. If it's better than we're currently doing, we might want to use it. So there's a very clear problem here that we need comparisons. And unfortunately, much of diagnostic research is comparison-free. And this is where actually we end up not having to be too troubled by publication bias, because if you don't make a comparison, you never need to conclude that your study has shown your test is useless. So it doesn't take 2 minutes to go to somewhere like Medline. This one was very quick. I just put in colon cancer and MRI, and I got a study here This is very typical of what we see in the literature. The aim of the study is to evaluate imaging. Here's the sensitivity, the specificity. These figures are a bit low, these ones are quite high, and the conclusion is it's useful. That's— a lot of our literature's like that. There's nothing in here— oh, that's a bit jumpy upstairs— to say— it's probably the author of the paper or something. But there is— I didn't want to pick on one paper, but you have to. There's lots of research like that. It doesn't take 2 minutes to find lots of examples which don't hesitate to draw conclusions. So what would we want on here? This is our plot of pulse oximetry. We need our compared to what on here. What are we going to put on there? Well, that's our pathway from our study. Standard practice is at the bottom that actually we don't do pulse oximetry, we just do a clinical examination. So we actually need to compare the accuracy of those 2 blue boxes. That's what we'd want to actually know whether adding pulse oximetry makes a difference. Unfortunately, you know, I don't have the data. We don't have the data on lots of these things as to how good the clinical examinations are, how good standard practice is. So this is a fictional point. This would be one thing, but that point, of course, is not the one to compare with because that's not pulse oximetry with clinical examination. This would answer the question about replacing pulse oximetry, replacing the clinical examination just with pulse oximetry. And we would see actually this would increase sensitivity and increase specificity. So that would be a good thing if this blue dot was true. We might think actually that is the one we want and it might be a little bit better. It might move around a bit which will give us a trade-off between sensitivity and specificity. So we need to think about what the comparisons are. And in the Cochrane Reviews we are trying to encourage our authors to do that, to say if this is about helping people make a management decision, Management decisions are choices. Give us the accuracy of the 2 choices and we can have a look at it. Now, there are sets of studies we can do which are really good for this. So you sometimes see them in literature. This is a multiple test comparison where all the patients get all the tests. So they get a routine test, they get the new test, they get the reference standard, and we can estimate the accuracy of either test and compare them. And because these patients are the same patients, it's a like-with-like comparison. And we can randomize patients to have one test or another test and all get the reference standard so we get accuracy. So these are robust ways we have of evaluating tests to actually make sure we can compare them. And if you've used those, you'll find in the Cochrane Library we're doing these strange graphs which don't really make it look any better or certainly not neater. But what we're doing is saying on these graphs we can connect together the studies. Which— so the results of the same study for the 2 tests. So that big square is linked to that circle. These are 2 tests of— from the same study. So they're a like-with-like comparison. And this is where we think we're getting our strong evidence from, that we actually are restricting studies we want to look at when we make comparisons to those which were designed to make comparisons. Unfortunately, this stuff is not very common. What we tend to do when we're doing evidence synthesis is to do one meta-analysis of the accuracy of one test and another meta-analysis of the accuracy of the other test. And at the end of the day, we compare the results of the meta-analyses. And we're happy to produce guidelines and recommendations based on this. This paper should be out in Annals next week. I'm not sure whether I was allowed to do that from the proof, but don't tell anybody. In this paper, we've actually looked at this to see how common is this being done and does it make any difference. And we found that we found 248 reviews which have compared tests. In those reviews, there were about nearly 7,000 studies. Less than a third of them were actually designed to compare tests. The rest of them were evaluating one or other of the tests in the comparison. Most of the studies in these reviews were non-comparative, nearly twice as many per review. And if we looked at these 248 reviews, only 11% of them were restricted to studies which were designed to compare the 2 tests. The rest either mixed together studies which were with those which weren't, or only found studies of one or other test. So this is the standard of our evidence. When we're saying, can we select which test? It's a bit poor. And when we look at, does this matter, and this is the first evidence we've— I think has been found to actually show that what we're doing is wrong, that actually we do need to design studies. There's significant evidence here that the differences we're seeing are a lot larger than we'd expect. There's no direction to them. It's not like you're always going to overestimate or underestimate. It's all a bit messy. But it's saying we're actually, by taking one set of studies to look at one test, comparing them to the other set of studies to look at the other, we're making— there is a bias. So are we guilty here of sloppy science? I think we are. We've had this very strong emphasis that when we compare treatments, we must randomize. We don't insist on it for tests. We don't even think about it for tests half the time. A lot of our evidence for tests is based on what we've seen. So reasons why this— has not been done? Well, I think there's problems here that there's no standard comparator sometimes. What do we compare with? Some people would use one thing as standard practice, other places it's another. I think that's actually a little bit of a poor excuse. It's logistically difficult, that's a very true study. In that pulse oximetry study we did, we didn't do a comparison. It was— we thought it was actually going to be too difficult to do. We thought about it long and hard. The reference standard might be the comparator sometimes. I think this is probably a lot of it is we're not really brave enough to try doing this. We're happy to shout the randomized trial banner for interventions. People know what we're talking about. But when I try and do it for tests, it's a hard slog with the clinicians to get them to say, yeah, I agree, we need to do a lot more work. We should be doing better. Okay, so the final bit. Try to make sure we can finish to get to the next session. It's not all about accuracy. Improved accuracy can lead to patient benefit if more patients receive effective treatments, but tests affect patients in other ways. So accuracy does need to be considered, but more than accuracy needs to be considered. One of my colleagues, Lavinia, has looked at a large number of trials of tests, and it's given us some insights into some of the ways in which tests impact on patients. This is a lovely example. where people did a trial of PET for staging non-small cell lung cancer. And in this trial, the idea is that PET is more accurate so you can identify those who don't need to go on and have a thoracotomy. But in— what actually happened in this trial was the PET was accurate, but none of those benefits from it were seen. 4% didn't have thoracotomy in one group compared to 2% in the other group. Why did that happen? Well, it happened because the surgeons, and they admit this in the discussion of this paper, didn't trust the PET results. They didn't quite know what to do with them. They were too new for them. And so there was no confidence in the test. So even though an accurate test existed in this trial, it went through this end-to-end study, and they found there was no benefit. That was a bit of a sad trial to do. Hopefully, that's not applicable. It's one of the reasons we don't always think about doing these trials is you have to have buy-in from the clinicians that doing— following the strategy of what you do with the PET is going to be agreed before you start. But let's look at a couple of other ways. I mean, these are very obvious ways in which tests affect patients. Why do we do sentinel lymph node biopsies for breast cancer? We can actually use the same diagnostic decision It can lead to the same treatments. It's nearly as accurate, but the main thing is about harms, that actually we're doing this test to avoid doing a more invasive test. And when we're looking at how well a test works, we need to think very clearly as to what— whether there's direct harm to the test. And a final example here, echocardiography, exercise ECG or stress echo. Well, the main benefit of stress echo we found in this trial was actually not in the accuracy so much, but it actually is in the feasibility of doing the test. A bit like one of those examples from the NICE appraisal committee is actually this test, the stress echo, is more feasible, can be used in more patients, and more often leads to conclusive diagnosis. The test is completed more often than an exercise ECG. So that decreases further testing. which hopefully will have its impact on patient benefit. So how do you identify all of those things? Well, we've been through all 100 trials and documented every different mechanism we could see. And this is reported in this paper which is cited in the abstract. And basically we found that when you thought through all of the different situations, we came up with a total of 14 different ways in which tests— impact on patients. We've not got time to go through them. You can read the paper, but they fall under headings of decision-making, which includes confidence as well as the test accuracy, timing of tests. Early tests can lead to patient benefits if treatment, early treatment is more effective. And the direct effects, the acceptability, the direct harms, and the feasibility. And there are lots of instances where we saw some slightly unpredictable effects. Patient and clinician factors and preferences impacting quite strongly on how a test result affects outcomes. So in summary, diagnostic evidence needs to assess tests. Are they used in pathways? It needs to think very strongly about making comparisons and plan to do primary research which makes comparisons. And it needs to think about how do we actually evaluate all of those aspects of how tests impact on patients. So reviews are difficult to answer that question, is does this review tell me that this test works, because they don't contain a lot of this information. And the primary research often hasn't been done to show these issues. We might say that test treatment RCTs where we randomize patients are the right way to go, and we look for outcomes, we look for the outcomes, but our looking at these 100 trials suggests they often don't deliver. They face many challenges and very few of them are ever done in diagnosis. They work very well in screening, but they don't work so well in diagnosis. So in summary, alternative evaluative approaches like these linked evidence approaches, I would suggest actually we call them portfolios of evidence because we might need to know a little bit more about some issues such as patient preferences which are hard to build into these linked models. That they may be a way in which we get a fuller, better analysis. So reviews of diagnostic accuracy are a necessary part of this. We always need to know how well our— how good our decision-making could be, but by a long way they're not sufficient to answer all of our questions. Thank you. Thank you.