PROGRESS in Prognosis Research

Hi there, I'm Richard Riley. I'm a professor of biostatistics at Keele University in the UK. In this short video, I'm going to introduce the PROGRESS framework to improve the quality of prognosis research, and I'll begin by explaining what we mean by prognosis and what we mean by prognosis research. So what is prognosis research? So prognosis is from the Greek foreknowing, foreseeing. It's the study of the risk of future outcomes and events. So we're looking forward in time to understand the chance of certain things of interest occurring in the future. And we're all interested in making prognosis, and we do this repeatedly every day of our lives. What's the weather going to be like today? Will it rain? Will my favorite sports team win a trophy this year? Will my baby give me a good night's sleep? So in healthcare, the term prognosis refers to the risk of future health outcomes, often adverse outcomes like death, in people with a given disease or health condition of interest. So for example, it may be in people who have just been diagnosed with cancer, or maybe in women who have just become pregnant, and we want to understand their likely outcomes and their risk Often of adverse events happening such as mortality, preeclampsia, and other things like that. So prognosis research is this study, the investigation of those relevant outcomes in those people with that given particular health state of interest. In other words, we're interested in start points— diagnosis disease, onset of pregnancy— and endpoints. things that haven't happened yet, but things that may occur in the future. So the research is trying to understand, summarise, and predict outcomes in those clinically relevant populations and individuals. Clearly, we want to inform clinical research and improve clinical practice, but the ultimate aim is to improve the health and well-being of populations and individuals and to reduce the risk of adverse outcomes from occurring. Is prognosis research relevant? Clearly it is. Indeed, it's probably never been more relevant because more and more of us are living with not just one health condition but multiple health conditions and comorbidities at a given time. And governments and funders, when they invest in prognosis research, they want to see that it's improving patient outcomes. That's the impact, that's the ultimate aim. Of research funding. When Jeremy Hunt was the UK Secretary of State for Health, he made it clear that the research needs to improve health outcomes, that we need to reduce variation in outcomes across the country to make different regions be more consistent with other regions, and we need to ensure that our outcomes as a country in the UK, for example, are not below those of other countries. This is evidenced by the NHS Commissioning Board document that was published a few years ago, which outlined how about £80 billion per year was to be spent, and the red thread was outcomes. I mean, look at some of these quotes: The government has set out a clear vision for a modernised NHS driven by a new commissioning system focused relentlessly on improving outcomes for patients. The purpose of the Board will be to use the budget to secure the best possible outcomes for patients. All about outcomes. Prognosis research can impact improved health outcomes at multiple pathways towards healthcare delivery. So if we look at this translational pathway here, we see we start from very basic research ending up at healthcare delivery, and prognosis research is relevant across all these different subcomponents. For example, at the start in basic research, we may use prognostic factors to help define our disease, define subtypes of disease. In terms of discovery, prognostic factors may help us to identify targets for new treatments. In clinical trials, prognostic information can be used to inform sample size calculations. Which are needed, for example, in terms of the baseline risk. In health technology assessment, we often are developing, validating, assessing prognostic models for individualised risk prediction. And then delivery— how do we use all those things to prioritise treatments for those at the highest risk, to implement precision medicine where we're targeting treatments based on particular genetic profiles, Prognosis research is hugely relevant. I now want to introduce to you the PROGRESS framework. So PROGRESS refers to the Prognosis Research Strategy, and the PROGRESS group relates originally at least to people who met unfunded for about 3 years but all had the same heart to improve the quality of prognosis research. And it was led by myself, Harry Hemingway at UCL, and Doug Altman at Oxford, and we'd all observed over many years the problems of prognosis research in terms of poor analysis, poor study design, poor reporting, general poor quality that led us to be very worried about the field. Ultimately, we were able to get an MRC partnership grant to undertake various types of initiatives to improve prognosis research. Here are some more details of members of the PROGRESS partnership. Now, our main output was indeed the PROGRESS framework. So this relates to a series of 4 articles that were published by the BMJ and PLOS Medicine in 2013, and this established or set out 4 strands for different types of prognosis research: overall prognosis, prognostic factors, prognostic models, and predictors of treatment effect. Now these themes are not necessarily distinct from each other, but they generally move from more population-level prognosis, average prognosis, all the way down to more individual-level predictions based on multiple pieces of information, either in terms of absolute risk or in terms of treatment effect prediction. So I just want to go through these 4 types of prognosis research as defined by the PROGRESS framework. The first one is overall prognosis, and this aims to describe and explain future outcomes in relation to current clinical care. So in the context of current diagnostic and monitoring strategies and treatments available, what is the overall average prognosis in patients with the condition of interest? Also, it's known as fundamental prognosis. Outcomes research is a popular name. Baseline risk, natural history. The aim is to reflect on the quality of current healthcare, to identify patterns of care and the overall impact of the current clinical care on future outcomes. So we can reveal the average outcome risk for a given health condition and the variations in that risk, for example, across different countries, across different regions, across different settings. It provides an initial answer to the question, what is the prognosis of people with this particular given disease or health condition? Clearly, if a population has a poor prognosis, then that motivates further inquiry in terms of why have they got that poor prognosis and what can we do about it? What money needs to be invested to improve outcomes in those people? A very nice but simple example of overall prognosis research is here. So in 2006, an average of about 15% of people aged over 65 and admitted to a US hospital with a heart attack died within 30 days. But that was compared to 19% in 1995. So over those about 10 years, there's been an improvement of about 4%. 4% less people are dying with a heart attack within 30 days. So that tells us perhaps that our standards of care has improved. This graph also demonstrates that as well. So we've got the 2006 in the bluey-green and we've got the 1995 in the white, and I'm plotting here a histogram of the number of hospitals against the 30-day mortality risk. The number of dead by 30 days for every 100 admitted with a heart attack. And we see that the, the mean of the 2006 histogram is 15%, and the mean of the 1995 histogram is about 19%. So the distribution has shifted to the left over the 10 years, which is good— mortality risk is reduced. But also look at the variability. The variability in 2006 across these different hospitals is much less than it was in 1995, where the variability is much wider. So that suggests that not only we're doing better on average, but also the variability is reduced, the outcome risks are much more consistent across the different hospitals, which is suggesting perhaps that we're standardising care better than we were 10 years ago. Another example here is the average 5-year survival of breast cancer patients displayed across different countries, and this is provided by Cancer Research UK in 2009. And we see that in Iceland they've got about a 90-95% chance of being alive after 5 years, but in the Czech Republic about a 70% chance. So again, this motivates further inquiry. Why are there differences? Is it due to different screening programs, different times of being diagnosed, maybe that in Iceland breast cancer patients are being diagnosed earlier and that's why their prognosis is better? Or is it to do with different standards of care and quality of care? Prognostic factor research aims to start to refine the overall prognosis summaries that I've just shown you in Type 1. So a prognostic factor is any measure that among people with a given health condition of interest, the factor is associated with the risk of a subsequent clinical outcome. Often we have very basic factors that are prognostic, such as age and weight and family history. Increasingly, we're interested in novel factors such as biomarkers in terms of whether they're prognostic. Different levels or values or groups of a prognostic factor are associated with a different average prognosis. Here's a very traditional example: tumor grade in breast cancer patients. Here I've got a Kaplan-Meier curve, and we can see that those in the 3rd tumour grade are at a higher probability of recurrence of disease or death over time than those in the lowest tumour grade, tumour grade 1. The hazard ratio comparing 3 to 1 is 3.63. There's strong evidence that tumour grade is a prognostic factor. So different levels of tumour grade are associated with different average risks. There are many potential benefits and uses of prognostic factors, from defining disease at diagnosis to informing clinical and therapeutic decisions. Ultimately, they may be used within a multivariable prognostic model, and we'll see that in a moment. And they may even, under a causal assumption, be targets for new treatments. But current prognostic factor research is riddled with problems. There's so many of these studies, and there's so many that are poor quality, sadly. For example, they have poor analysis methods, they dichotomize continuous variables and outcomes. Publication bias is a major problem where people are only publishing those factors that they found to be, let's say, statistically significant after lots of data dredging and lots of searching. So replication is a problem in this field. Another major issue is that people generally forget the basic factors and don't look at the added value of a new novel factor over and above standard factors. Everyone's searching for something new rather than trying to confirm what we already know about. Prognostic models utilize multiple prognostic factors in combination, and the aim is to predict the risk of future outcomes of interest at the individual level. So we also know these prognostic models by names such as multivariable prediction models, And often there's a statistical equation behind the scenes such as logistic regression or Cox regression. Increasingly, machine learning methods are being used to develop these models. The aim is to provide individualized risk predictions that inform patients and their caregivers. So understanding risks at the individual level is crucial for shared decision-making and to plan for the future. Clinical research such as randomized trials can utilize prognostic models in their trial design and randomization, for example. And ultimately, understanding somebody's risks can help us make decisions about what treatments to implement for that individual. So prognostic model research moves away from average group-level prognoses and focuses more on individual risk for individuals. Let's look at a real example. This is a prognostic model called CRASH, and this is for use in patients with a traumatic brain injury. So patients have a traumatic brain injury, they get admitted to hospital, and we could use this tool to make predictions to assess the risk for this individual. This tool needs you to enter values of prognostic factors that are in the model, which is actually a logistic regression behind the scenes. Here it's country, age, Glasgow Coma Score, reactivity to light, whether there was extracranial injury, etc. We enter the values for this individual patient, and the model behind the scenes would give us a risk of 14-day mortality, or— and also the risk of unfavorable outcome at 6 months. So for 14-day mortality, it's 14%, and for 6 months, it's 50%. Prognostic model research has 3 main phases: model development, where we develop the model equation and we minimize overfitting and do some internal validation. Then there's external validation, where we look at the performance of our model in new data, often in a different population or setting to that that was used to develop the model. And then the third one is to investigate the impact. So if we use the model in practice, then is it likely to lead to better outcomes than if we didn't use it? Again, although there's so much potential here, Current research focuses mainly on model development, and there's, there's big problems. So there's often small sample sizes, people don't adjust for overfitting, people don't do internal validation. And then even moving on to external validation, we have problems such as these studies are rarely done, people are more interested in developing their new model than validating an existing model, again small sample sizes, and Calibration of predictions is rarely assessed, many, many other issues. And then impact studies are extremely rare. So for example, randomized trials evaluating the use of a model versus not using it are sadly rare. The 4th progress type is predictors of treatment effects. This is where we're trying to look at which treatments work well for particular people. So traditionally we have an approach where we'd look at the average of effect of a treatment across all people in a particular population, but it could be that some treatments work for some and not for others. And if we can understand why that is, then we can tailor treatment to maximize treatment-related benefits and to reduce treatment-related harm. A key part of this type of research is to identify those predictors, sometimes known as tests, and these are often biomarkers or genotypes that actually do predict the treatment effects at the individual level. So if we know a characteristic of an individual that will tell us whether a treatment will work or not, then we can help healthcare professionals in their decision-making. Here's a probably well-known example. Trastuzumab has been shown to be more effective in those HER2-positive breast cancer patients. We see the results of this trial where the test-negative individuals, the relative risk is much closer to 1 than those that are test positive, where there's very strong evidence of an effect. And therefore, it could be that we focus more on the use of this treatment in those that test positive than test negative, and there's biological rationale for that. But as for the other 3 sites of prognosis research, there are flaws in this area, and it's sad to say, but it's true. And actually, we worry that the evidence to support many predictors of treatment effects is quite weak. The field is riddled with problems such as poor study design, poor analysis methods, and incomplete reporting. And this leads to spurious evidence. So small sample sizes, for example, lead to low power. So there may be some genuine predictors of treatment effects out there that we just don't know about because we haven't powered our studies properly. More common is data dredging, where people keep looking for subgroup effects and only report the ones that they found to be significant. So this leads to lack of replication and new data. Dichotomization of continuous variables and outcomes is a major problem, and you could even argue on the previous slide that, you know, the dichotomization of HER2 into positive and negative may not be appropriate. Lack of adjustment for prognostic factors and other variables. But to finish, I want to focus on improving prognosis research. So we've introduced the PROGRESS framework, we've introduced some of the issues in the field, but let's focus on the positives. Some pointers to improve prognosis research if you're thinking about setting up your own prognosis research study: really think about the clinical context. Okay, obviously what is the start point? What is the health condition of interest? But also, at what time do you want to implement your prognostication? Is it a diagnosis of disease? Is it a particular follow-up point? Is it after treatment's been initiated? And what are the endpoints of interest? What are the outcomes? What you're trying to study? What you're trying to improve? What is relevant to the doctors? What's relevant to the patients? And in that context, what prognosis questions are you interested in? Where does your research fit in the PROGRESS framework? Are you interested in prognostic factors, or are you interested in predictors of treatment effect? People often get those 2 things muddled up. Are you interested in developing a prognostic model? What previous evidence is out there? This is a really big question to ask yourself. There's no need to develop another prognostic model when there's already 100 out there. Why not maybe try and externally validate some of those existing models? Maybe they're already good enough. If you're looking at a novel factor for its prognostic ability, then does it actually add value over and above standard easily measured variables that we know are prognostic already? So build on previous research. When we're designing new studies to answer our questions, let's write our protocols, let's do a sample size calculation, let's use appropriate statistical methods, let's report our results transparently. So the drive to improve prognosis research through the PROGRESS framework ultimately led a number of us to produce this book, Prognosis Research in Healthcare. In this book, we talk about the key concepts of prognosis and prognosis research and how to design studies looking at each of the 4 prognosis types: overall prognosis, prognostic factors, prognostic models, and predictors of treatment effect. We give clear instruction to statistical methods and terminology. We even move beyond primary studies into systematic reviews, big data, and IPD meta-analyses, and have some clear illustrated examples in various chapters of impact of prognosis research in some disease fields. There's even a chapter on machine learning and the potential of machine learning to help with developing prognosis and prognostic models. The book includes contributions from a number of leaders in this field including Doug Altman, Gary Collins, Kim Snell, Thomas de Bray, Harry Hemingway, Ewart Starberg, amongst many others. The main drive for this book and indeed to improve prognosis research stemmed from Professor Doug Altman. So Doug was way ahead of his time. Whereas people were talking about how to improve trials and diagnostic tests, Doug in the 1990s, for example, was already talking about the poor quality of prognosis research studies. And this book was dedicated to Doug and all that he stood for, and we will continue to drive forwards and improve prognosis research on his behalf. If you want to learn more about Doug's work in prognosis research and indeed many, many other areas, I can highly recommend this paper by Billy Saalbrecht et al., published in 2020 in the Biomedical Journal: Doug Altman: Driving Critical Appraisal Improvements in the Quality of Methodological and Medical Research. Alongside the PROGRESS framework, there's been many other positive initiatives. For example, reporting guidelines have come into play. For the REMARC for prognostic factors and TRIPoD for prognostic models. Quality appraisal tools, for example, QUIPS for factors and PROBUS for prediction models. STRATOS, a group of methodologists aiming to identify good standards and good practice for statistical methods and reporting in observational studies. Guidance has been released to improve sample size calculations for prognostic factors and prognostic model research. Cochrane have started implementing systematic reviews and meta-analysis of prognosis studies, so we can actually review and establish quality and reliability of the current evidence in particular fields. We have now got dedicated training courses, conferences, and symposiums in prognosis research, and our website prognosisresearch.com details those. Our website is an entry-level website to prognosis research. We aim there to promote good practice, to disseminate the PROGRESS framework, but also to give an introduction, a framework, a guide to doing prognosis and doing it well. So thank you for listening. If you would like further details or to give feedback, then please do get in touch with me and do visit our website for further information.