Abstract
Background:
Clinical guidelines commonly recommend preventative treatments for people above a risk threshold. Therefore, decision-makers must have faith in risk prediction tools and model-based cost-effectiveness analyses for people at different levels of risk. Two problems that arise are inadequate handling of competing risks of death and failing to account for direct treatment disutility (i.e. the hassle of taking treatments). We explored these issues using two case studies: primary prevention of cardiovascular disease using statins and osteoporotic fracture using bisphosphonates.
Objectives:
Externally validate three risk prediction tools [QRISK®3, QRISK®-Lifetime, QFracture-2012 (ClinRisk Ltd, Leeds, UK)]; derive and internally validate new risk prediction tools for cardiovascular disease [competing mortality risk model with Charlson Comorbidity Index (CRISK-CCI)] and fracture (CFracture), accounting for competing-cause death; quantify direct treatment disutility for statins and bisphosphonates; and examine the effect of competing risks and direct treatment disutility on the cost-effectiveness of preventative treatments.
Design, participants, main outcome measures, data sources:
Discrimination and calibration of risk prediction models (Clinical Practice Research Datalink participants: aged 25–84 years for cardiovascular disease and aged 30–99 years for fractures); direct treatment disutility was elicited in online stated-preference surveys (people with/people without experience of statins/bisphosphonates); costs and quality-adjusted life-years were determined from decision-analytic modelling (updated models used in National Institute for Health and Care Excellence decision-making).
Results:
CRISK-CCI has excellent discrimination, similar to that of QRISK3 (Harrell’s c = 0.864 vs. 0.865, respectively, for women; and 0.819 vs. 0.834, respectively, for men). CRISK-CCI has systematically better calibration, although both models overpredict in high-risk subgroups. People recommended for treatment (10-year risk of ≥ 10%) are younger when using QRISK-Lifetime than when using QRISK3, and have fewer observed events in a 10-year follow-up (4.0% vs. 11.9%, respectively, for women; and 4.3% vs. 10.8%, respectively, for men). QFracture-2012 underpredicts fractures, owing to under-ascertainment of events in its derivation. However, there is major overprediction among people aged 85–99 years and/or with multiple long-term conditions. CFracture is better calibrated, although it also overpredicts among older people. In a time trade-off exercise (n = 879), statins exhibited direct treatment disutility of 0.034; for bisphosphonates, it was greater, at 0.067. Inconvenience also influenced preferences in best–worst scaling (n = 631). Updated cost-effectiveness analysis generates more quality-adjusted life-years among people with below-average cardiovascular risk and fewer among people with above-average risk. If people experience disutility when taking statins, the cardiovascular risk threshold at which benefits outweigh harms rises with age (≥ 8% 10-year risk at 40 years of age; ≥ 38% 10-year risk at 80 years of age). Assuming that everyone experiences population-average direct treatment disutility with oral bisphosphonates, treatment is net harmful at all levels of risk.
Limitations:
Treating data as missing at random is a strong assumption in risk prediction model derivation. Disentangling the effect of statins from secular trends in cardiovascular disease in the previous two decades is challenging. Validating lifetime risk prediction is impossible without using very historical data. Respondents to our stated-preference survey may not be representative of the population. There is no consensus on which direct treatment disutilities should be used for cost-effectiveness analyses. Not all the inputs to the cost-effectiveness models could be updated.
Conclusions:
Ignoring competing mortality in risk prediction overestimates the risk of cardiovascular events and fracture, especially among older people and those with multimorbidity. Adjustment for competing risk does not meaningfully alter cost-effectiveness of these preventative interventions, but direct treatment disutility is measurable and has the potential to alter the balance of benefits and harms. We argue that this is best addressed in individual-level shared decision-making.
Study registration:
This study is registered as PROSPERO CRD42021249959.
Funding:
This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 15/12/22) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 4. See the NIHR Funding and Awards website for further award information.
Plain language summary
Before offering a medicine to prevent disease, prescribers must expect it to do more good than harm. This balance depends on how likely it is that the person will develop the disease we want to prevent. But people might first die for other reasons. We call this a ‘competing risk’. In most cases, the mathematical tools we use to estimate the chance of developing a disease do not account for competing risks. Another problem is that, when weighing up the benefits and harms of medicines, we ignore the hassle they cause patients, even when they do not cause side effects.
We used two examples: statins to prevent heart disease and bisphosphonates to prevent fractures. First, we assessed if existing tools get predictions wrong by not accounting for competing risks. We found that they exaggerate the chance of heart attacks and strokes. However, the exaggeration is greatest among people who would clearly benefit from preventative treatment. So it may not change treatment decisions much. The fracture prediction tool we studied was very inaccurate, exaggerating risk among older people, but underestimating risk among younger people. We made a new fracture risk prediction tool. It gave better predictions, but it was still inaccurate for people aged > 85 years and those with several health problems.
Next, we asked people questions designed to put a number on the hassle that statins and bisphosphonates cause. Most people thought that taking either is inconvenient, but the hassle factor for bisphosphonates is bigger.
Finally, we updated the mathematical models that the National Institute for Health and Care Excellence used when recommending statins and bisphosphonates. We worked out if competing risks and the hassle of taking medicines make a difference to results. Statins remain a good idea for almost everyone, unless they really hate the idea of taking them. But bisphosphonates would do more harm than good for anyone who agrees with the hassle factor we found.
About the Series
Health and Social Care Delivery Research
ISSN (Print): 2755-0060
ISSN (Electronic): 2755-0079
Full disclosure of interests: Completed ICMJE forms for all authors, including all related interests, are available in the toolkit on the NIHR Journals Library report publication page at https://doi.org/10.3310/KLTR7714.
Primary conflicts of interest: Sarah Davis has received a grant from Roche Diagnostics Ltd (Burgess Hill, UK) to examine the cost-effectiveness of bone turnover markers to monitor osteoporosis treatment in primary care.
Article history
The research reported in this issue of the journal was funded by the HSDR programme or one of its preceding programmes as project number 15/12/22. The contractual start date was in September 2015. The final report began editorial review in February 2022 and was accepted for publication in June 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HSDR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Last reviewed: February 2022; Accepted: June 2022.
