Handling uncertainty when performing economic evaluation of healthcare interventions
Briggs AH., Gray AM.
Aims: To perform a structured review of the way in which uncertainty has been handled in economic evaluation. To assemble data on the actual distributional form and variance of healthcare costs, and to devise guidelines to improve current practice. In particular, the focus was on the handling of cost and cost-effectiveness data. Methods: The structured review was conducted at a number of different levels, reflecting the detail of the review process. At a general level, a search of the literature was undertaken to identify published economic evaluation studies that reported results in terms of cost per life-year or cost per quality-adjusted life-year values. This form of study was chosen as it is the results of these studies that are commonly grouped together and reported in cost-effectiveness league tables. Articles meeting the search criteria were reviewed using a review proforma designed to collect summary information on each study. These results were then entered as key words into a database, to allow interrogation and cross- referencing of the database by category. This overall data set was then employed to focus in on two specific areas of interest, using subsets of articles to perform more detailed reviews: All studies reporting UK results were identified from the wider group of articles. These studies were reviewed in detail, and information on the baseline cost-effectiveness results, the methods underlying those results, the range of results representing uncertainty and the number of previously published results quoted in comparison were entered into a relational database. By matching results by the methods employed using a retrospective application of a methodological 'reference case', a subset of results with improved comparability was identified, and a rank ordering of these results was then attempted. Where a range of values accompanied the baseline results, the implications of this uncertainty for the rank ordering was also examined. All studies which reported patient level cost data were identified from the overall database and reviewed in detail with respect to how they had reported the distribution and variance of healthcare costs. In addition, five available data sets of patient level cost data were examined in order to show how the healthcare costs in those data were distributed and to elucidate issues surrounding the analysis and presentation of healthcare cost differences. Economic analyses are not simply concerned with costs but also with effects, with the cost- effectiveness ratio being the outcome of interest in most economic evaluations. Unfortunately, ratio statistics pose particular problems for standard statistical methods. In this report, a review of a number of proposed methods for estimating confidence limits for cost-effectiveness ratios when patient level data are available for both cost and effectiveness is presented. Results: A total of 492 articles were found to match the search criteria, and were fully reviewed and entered into the database. Analysis of this database in terms of the method employed by analysts to handle uncertainty shows that the vast majority of studies use one-way sensitivity analysis methods only. Of some concern is that 17% of studies did not attempt any analysis to examine uncertainty, although there is weak evidence to show that this situation is improving. Of these 492 studies, 60 reported results for the UK. From these UK studies, 548 baseline cost-effectiveness results were extracted relating to 106 methodological scenarios. Application of a retrospective 'reference case' gave a single methodological scenario for each article with 333 associated baseline results. These results were converted to a common cost base year, and rank ordered to give a comprehensive 'league table' of UK results. Of the 333 results, 61 had an associated full range of values to represent uncertainty. Alternative rankings based on the high or low values from this range showed that there could be considerable disruption to the rank order based on the baseline point estimates only. The review of patient level cost data showed that 53 of the 492 studies in the database had patient level cost data and that just 15 of these had reported some measure of cost variance. Only four studies had calculated 95% confidence intervals for cost. The review of five available cost data sets showed that the cost data were not normally distributed, and in two cases showed substantial skewness. A number of methods for estimating confidence intervals for cost- effectiveness ratios have appeared in the recent literature. Examination of their statistical properties and evidence from recent Monte Carlo simulation studies suggests that many of these methods may not perform well in some circumstances. The parametric method based on Fieller's theorem and the non- parametric approach of bootstrapping produced consistently the best results, and are the preferred methods for estimating confidence intervals for cost- effectiveness ratios. However, the use of cost-effectiveness acceptability curves may provide more useful information to decision makers than standard confidence intervals. Conclusions. General recommendations: Potential guidelines arising from this review are: analysts should aim to present results using a methodological reference case in order to increase the comparability of results between studies; analysts should be aware of the potential for the incremental cost-effectiveness ratio to vary at the margin; analysts should avoid selective comparison of their results with the results from other studies; analysts should ensure that they consider the potential implications of uncertainty for the results of their analysis; interval estimates should accompany each point estimate presented; where sensitivity analysis is employed to estimate an interval, analysts should be comprehensive in their inclusion of all variables in the analysis; when reporting sensitivity analysis, analysts should be aware of the probabilistic nature of the reported range; when reporting patient level cost information, analysts should make more use of descriptive statistics; even when data are skewed, economic analyses should be based on means of distributions; when reporting statistical tests of cost differences, analysts should be aware that significance tests may be more powerful on a transformed scale but that confidence limits should be reported on the original scale; where patient level data on both cost and effect are available, the parametric approach based on Fieller's theorem or the non-parametric approach of bootstrapping should be employed to estimate a confidence interval for the cost- effectiveness ratio; sensitivity analysis has a continuing role in handling uncertainty not related to sampling variation; consideration should be given to using cost-effectiveness acceptability curves to present uncertainty in stochastic cost-effectiveness studies. Recommendations for future research: Three main areas for future research arise from this review: research into the appropriate reference case for the UK; research into the application of probabilistic sensitivity analysis methods; research into the willingness to pay for health gain and the likely value of a ceiling cost-effectiveness ratio appropriate for decision making, estimated from consumer surveys and implied through the application of cost-effectiveness databases.