For citation purposes: Ramaekers B, Grutters J, Joore M. Acknowledging patient heterogeneity in health technology assessment. OA Epidemiology 2014 Feb 10;2(1):3.



Acknowledging patient heterogeneity in health technology assessment

B Ramaekers1*, J Grutters2, M Joore1

Authors affiliations

(1) Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), Maastricht University Medical Center, P.O. Box 5800, 6202 AZ, Maastricht, The Netherlands

(2) Departments for Health Evidence & Operating Rooms, St. Radboud University Medical Center, Nijmegen, The Netherlands

* Corresponding author Email:



Patients often respond differently to identical treatments despite a similar diagnosis. Therefore, treatments in health care are increasingly tailored to individual patients. Nevertheless, this patient heterogeneity is often ignored in economic evaluations and hence in policy decisions based on these economic evaluations; whereas, this could potentially lead to a more effective and/or more efficient health care. This paper focuses on the role of patient heterogeneity in economic evaluation. One frequently mentioned barrier to acknowledging patient heterogeneity is a lack of suitable data. This could be overcome through an integrated modelling approach or on the longer term through risk-sharing agreements.


We have illustrated that it is feasible and informative to acknowledge patient heterogeneity in economic evaluation. One frequently mentioned barrier to acknowledging patient heterogeneity, a lack of clinical data, can be overcome by an integrated model using dose-response models or over a longer period of time by risk-sharing agreements and/or ‘open data’ policy for scientific journals.


Despite the high degree of similarities that bind humanity together as a species, considerable diversity exists between individual patients[1]. All patients are individuals and often respond differently to identical treatments despite a similar diagnosis[2]. This natural variation can be defined as variability. The part of this variability that can be explained by certain patient characteristics is defined as patient heterogeneity. These patient characteristics may include demographics (e.g. age, gender, income), preferences (e.g. attitude, beliefs, risk tolerance) and clinical characteristics (e.g. disease severity, disease history, genetic profile)[3]. Patient heterogeneity should be distinguished from treatment heterogeneity, which refers to differences in the nature of the treatment (e.g. differences in treatment dose or technique). Patient heterogeneity is increasingly acknowledged in clinical practice through anticipating on predictable differences in treatment response and aiming to tailor treatments to individual patients based on this information[4,5]. The promise of personalised medicine is that it will improve treatment efficacy, reduce toxicity and minimise costs[6]. This minimisation of costs can be considered valuable in the light of the dramatically increasing health care costs in Western societies and the growing attention on costs of expensive novel technologies. Considering these escalating costs, the subsequent rising insurance premiums and the finite health care budgets, available resources should be allocated as efficient as possible and choices between health care technologies have to be made. This implies restricting reimbursement, in the statutory package of insured care, to health care technologies that provide acceptable value for their money. Health Technology Assessment (HTA) is a field of research that aims to inform health policy makers in these decisions by examining the medical, economic, social and ethical implications of a medical technology in health care[7]. Within HTA, economic evaluations are increasingly employed to examine the costs of novel technologies in relation to its effects. Similarly as mentioned above for clinical practice, acknowledging patient heterogeneity in economic evaluations would be beneficial as it potentially improves the efficiency and/or effectiveness of health care[3]. Patient heterogeneity is, however, often neglected in economic evaluations[8]. This paper aimed to discuss the feasibility and relevance of acknowledging patient heterogeneity in economic evaluation.

Economic evaluation

Economic evaluation is a frequently used tool for HTA to compare the costs and consequences of different health care technologies. These consequences or effects are often expressed in terms of quality-adjusted life years (QALYs)[9]. The QALY is a product of the quantity and the quality of life lived. It is a combined measure that captures both life expectancy and generic health-related quality of life (HRQOL) in a single value. Ideally, economic evaluations consider all relevant evidence with regard to the costs and consequences and compare the full range of available treatment options. Moreover, model-based economic evaluations (i.e. decision-analytical models) are often required to incorporate all relevant evidence, estimate final outcomes rather than intermediate outcomes, consider all relevant comparators and extrapolate the results to a lifetime time horizon[10]. Decision-analytical models can act as an analytical framework to combine effectiveness data with other types of evidence such as quality of life data, data on resource use and unit prices[11]. Ultimately, all parameters in these model-based economic evaluations are based on a meta-analysis or other form of synthesis of available evidence[11]. Based on these data, decision-analytical models (predominately Markov models) aim to reflect the course of a disease, using a hypothetical cohort of patients who transit between mutually exclusive health states, to compare the costs and consequences of competing interventions[12]. More specifically, economic evaluations aim to inform two distinct but connected questions[13]:

1. whether current evidence suggests that the new treatment is cost-effective compared with current practice and

2. whether further research into this matter would be worthwhile.

The answer to the first question depends on the differences in costs and QALYs between the treatments of interest and the amount of money that society is willing to pay per gained QALY. This amount is referred to as the ceiling ratio. This is illustrated with an example considering a new treatment that leads to an average gain in QALYs of 0.100 and is on average €10,000 more expensive compared with current practice. These average incremental outcomes are presented (diamond) in Figure 1(a). The diagonal line represents a ceiling ratio of €80,000 per QALY gained. This is the informal ceiling ratio for a high burden of disease in the Netherlands[14]. As the diamond is above this ceiling ratio, this new treatment is not deemed cost-effective and should not be reimbursed based on this information. However, average cost-effectiveness estimates, as presented in Figure 1(a), are inevitably surrounded by uncertainty. As a result, it is possible that based on current information, the ‘wrong’ reimbursement decision is being made. Therefore, it is essential to characterise uncertainty in economic evaluations. This decision uncertainty is often incorporated in economic evaluations by reflecting the uncertainty in the input parameters of the economic evaluation representing that we do not know the exact estimates[11]. The eclipse in Figure 1(b) reflects this parameter uncertainty (or sampling uncertainty) surrounding the average cost-effectiveness. As illustrated in this figure, there is a substantial probability that the ‘true’ cost-effectiveness estimate for the new treatment falls below the ceiling ratio and would thus be cost-effective (Bayesian interpretation). This decision uncertainty is considered in the second question.

(a) incremental cost-effectiveness plane, (b) incremental cost-effectiveness plane with estimated parameter uncertainty surrounding the average cost-effectiveness and (c) incremental cost-effectiveness plane with individual cost-effectiveness.

The second question addresses whether further research would be valuable to reduce the decision uncertainty and (re-)inform the reimbursement decision in the future. This can be quantified by using the expected value of perfect information analysis to assess the expected costs of available decision uncertainty. This is assessed by considering the certainty that the new treatment is cost-effective and the consequences if adopting the new treatment appears the wrong decision[11]. The expected value of perfect information can be interpreted as the maximum amount society should be willing to pay to eliminate available decision uncertainty and inform the decision in the future.

The two questions mentioned above are usually informed by presenting the average cost-effectiveness for a group of patients[9,15]. Although parameter uncertainty is often considered in these economic evaluations (as in Figure 1b), variability and patient heterogeneity are frequently neglected in average cost-effectiveness estimates and subsequent reimbursement and research decisions[8]. Neglecting these genuine differences between patients might mask important variations in cost-effectiveness[15,16].

Acknowledging patient heterogeneity in economic evaluation

Neglecting patient heterogeneity in economic evaluations may lead to inefficiency. For instance, in Figure 1(c), it becomes clear that the average cost-effectiveness as presented in Figure 1(a) does not apply to all patients; the effectiveness and cost-effectiveness differ per individual patient. Based on average cost-effectiveness estimates for all patients, the new treatment might not be reimbursed for the total group, although there are subgroups of patients for whom it is expected to be cost-effective, potentially leading to a suboptimal allocation of available resources. Average population-based cost-effectiveness might also be suboptimal for reimbursement decisions if a treatment is not cost-effective for certain subgroups while these subgroups do receive this treatment since it is cost-effective on average. Again, this would lead to a suboptimal allocation of available resources. Therefore, a more individualised approach to the allocation of resources by providing and/or restricting treatment reimbursement to subgroups of patients (instead of one decision for all) has the potential to increase population health gains[15,16,17,18]. For instance, as illustrated in Figure 1(c), the new treatment is cost-effective on average for females (triangle below the ceiling ratio), whereas it is not on average cost-effective for males (triangle above the ceiling ratio). Therefore, acknowledging patient heterogeneity and thus reimbursing the new treatment for females and not for males would lead to a more optimal allocation of available resources compared with not reimbursing the new treatment for all patients. Moreover, acknowledging patient heterogeneity is considered as a quality criteria for good practice in economic evaluations[19]. Attempts have been made to acknowledge patient heterogeneity in economic evaluations by using stratified analysis[17], regression techniques[20,21,22] and/or estimating the expected value of individualised care[23] (note that these methods are not specific to model-based economic evaluations and can also be applied in trial-based economic evaluations). Despite these initiatives, patient heterogeneity is currently frequently neglected in economic evaluations and subsequent policy decisions[8]. This might in part be explained by a lack of guidance on how to acknowledge patient heterogeneity, the large demand of data, the computational burden associated with the analyses and equity concerns when excluding patient subgroups[24]. Preferably, the analyses are based on individual patient-level data[25]. In case these data are absent, average outcomes of comparative effectiveness research can be used. As comparative effectiveness research is often lacking in the early phase of new medical technologies, non-availability of data might be experienced as a serious barrier to the acknowledgement of patient heterogeneity in economic evaluation.

In case individual patient data and comparative effectiveness are lacking

Despite a potential lack of individual patient data or even comparative effectiveness research, it is possible to acknowledge patient heterogeneity in economic evaluation. This was, for instance, illustrated in the case of proton radiotherapy for head and neck cancer, for which comparative effectiveness research is lacking[26]. In this study, an integrated model was developed to overcome the lack of comparative effectiveness research[27]. This integrated model consisted of individual patient characteristics, a dose-response model and an economic model to assess the cost and consequences. Patient-specific radiation dose to the swallowing structures and parotid glands was predicted (model 1) and linked to dose-response models which predict the probability of dysphagia and xerostomia (model 2). Subsequently, these patient-specific complication probabilities were combined with HRQOL and cost data to estimate the effectiveness and cost-effectiveness for individual patients (model 3).

This integrated model could also act as a decision support tool to facilitate individualised decision making both for individualised treatment decisions in clinical practice (micro-level) and for individualised reimbursement decisions (macro-level). This methodology is based on surrogate outcomes and assumptions about reality and thus cannot be considered as a substitute for prospective clinical studies, rather a supplement. The methodology offers a solution if it is not desirable to wait or postpone decisions until clinical data become available. This can be the case, for example, in an early stage of a technology or if randomised trials are considered unethical or unfeasible. Moreover, as with decision-analytical modelling in general, the integrated model might be employed to enrich, extrapolate and broaden the results from available evidence by combining various evidence sources and types of analytical models[28]. Hence, even if comparative effectiveness research is available, the integrated modelling methodology might be used to address policy relevant questions[10].

Access to individual patient data

In case individual patient-level data are available, patient heterogeneity can be incorporated by using regression models to estimate input parameters dependent on patient characteristics (e.g. see Ramaekers et al.[29]). Individual patient-level data are preferred above surrogate outcomes (as in the abovementioned study) as these contain final outcomes and the highest grade of evidence[30]. Additionally, individual patient-level data provide more opportunities to identify subgroups to be considered in the analysis, to explore patient heterogeneity based on subgroups (e.g. to examine for which subgroups a treatment is cost-effective or additional research is valuable) and to determine the optimal number of subgroups.[25]Thus, to acknowledge patient heterogeneity and move towards personalised medicine, access to individual patient-level data is essential. Ideally, trial data should be made publicly available for this purpose. It can be questioned who should have the right to access the individual patient-level data. Data on what works in clinical practice can be considered a public good, independently of the entity (public or private) responsible for collecting the data[31]. When considering the massive amount of public money, and public trust which are placed in medical products that are adopted in clinical practice, it seems reasonable to request access to the full data[32]. It can even be argued that society should have access to these data since the society pays for industry-funded trials through patents. To stimulate this openness of data, governments should establish laws that go beyond current legislation to demand access to the individual patient-level data behind any analyses used to license or reimburse a medical product[32]. Additionally, scientific journals may facilitate this by using ‘open data’ policy as for instance now applies toNaturejournals: ‘authors are required to make materials, data and associated protocols promptly available to readers without undue qualifications’[33]. Ultimately, as put forward by others, this procedure should become the norm: ‘required by journals and accepted by the scientific community as mandatory’[34]. An ‘open data’ approach is the best way to maximise the benefits of research[34,35].

In addition to ‘open data’ policies, Performance-Based Risk Sharing Agreements (PBRSA) might provide opportunities to achieve data necessary for individualised decisions. Decision makers often face considerable uncertainty about effectiveness and cost-effectiveness of novel medical products and growing financial risks (due to the increasing costs of medical interventions). Partly in response to these concerns, the interest in PBRSA is substantial and rising[31]. PBRSA are arrangements, often between the payer and producer of a medical product, mainly aimed at reducing the uncertainty through investing in the collection of evidence, whereas a technology is temporarily adopted within clinical practice. These agreements concern a specified period of time and the (future) price and/or utilisation of the product is dependent on the outcomes achieved[31,36]. In this way, PBRSA may lead to value-based pricing and facilitate patient access to (high costs) drugs or technologies. Part of the decision uncertainty aimed to resolve in PBRSA may originate from patient heterogeneity. These arrangements may thus provide an excellent opportunity for decision makers to request the collection of data necessary to examine the effectiveness and cost-effectiveness for subgroups of patients, for instance, identified in previous trials or based on biological plausibility. This may lead to ex ante reimbursement schemes, where the treatment is only reimbursed for a selection of individual patients, for instance, based on a biomarker[31]. As an example, gefitinib for treatment of non-small cell lung cancer is only recommended by National Institute for Health and Clinical Excellence for patients with a positive epidermal growth factor receptor mutation status and based on an agreed price[37]. Alternatively, if it is not possible to identify subgroups before treatment initiation, PBRSA schemes focused on ex post reimbursement might prove valuable. This may involve schemes as ‘outcomes guarantees’ meaning that the product will only be (fully) reimbursed for patients who respond to treatment or ‘conditional treatment continuation’ meaning reimbursement for the continued use of a treatment based on intermediate outcomes[31]. These arrangements seem promising as they potentially result in a number needed to treat of one. However, these ex post schemes require a measure for response or an intermediate outcome that is feasible, valid and can be measured relatively shortly after treatment initiation.

The data gathered through ‘open data’ policy or PBRSA schemes may facilitate the acknowledgement of patient heterogeneity in decision-analytical models. This may involve complex methods[17,20,21,22,23] and as reimbursement decision makers are often not specialised in health economics, the cost-effectiveness results might become increasingly difficult to communicate. Therefore, to communicate the potential value of stratified or individualised decision making and influence reimbursement policy, these complex methods should be translated to less complex outcomes. In the abovementioned analysis considering proton therapy, the expected value of individualised care was translated by using an additional comparator next to ‘proton radiotherapy for all patients’ and ‘photon radiotherapy for all patients’. This additional strategy was named ‘proton radiotherapy if efficient’ and implies that patients for whom proton radiotherapy is expected to be cost-effective receive proton radiotherapy, whereas the remaining patients receive photon radiotherapy. The results of this strategy were presented as a ‘normal’ comparator using less complex outcomes such as the incremental cost-effectiveness ratio.


To increase the efficiency of health care and bridge the gap between individualised decisions in clinical practice and subsequent population-based reimbursement decisions, it is encouraged that acknowledging patient heterogeneity becomes standard practice in future economic evaluations. Although this is preferably based on individual patient-level data, the unavailability of these data does not justify neglecting patient heterogeneity. Acknowledging patient heterogeneity in economic evaluation is both feasible and informative, even if comparative effectiveness research is scarcely available. The international health economic community should develop consensus based on how available methods such as stratified analysis,[17] regression techniques[20,21,22] and the expected value of individualised care[23] should be used to acknowledge patient heterogeneity in the economic assessment of technologies. International societies, in particular the International Society for Pharmacoeconomics and Outcomes Research, are encouraged to adopt a leading role in this process. On the other hand, national decision makers are encouraged to provide specific recommendations in their pharmacoeconomic guideline when it is (ir)relevant (e.g. which characteristics can potentially be used for policy decision) to acknowledge patient heterogeneity in economic evaluations and what is required to inform their appraisal process.


We have illustrated that it is feasible and informative to acknowledge patient heterogeneity in economic evaluation. One frequently mentioned barrier to acknowledging patient heterogeneity, a lack of clinical data, can be overcome by an integrated model using dose-response models or over a longer period of time by risk-sharing agreements and/or ‘open data’ policy for scientific journals. Additionally, entities (public and private) responsible for collecting data are encouraged to make their data publicly available. This seems reasonable when considering the massive amount of public resources and trust that are placed in medical products adopted in clinical practice. Governments may facilitate this process by requesting openness of data for submissions during the licensing and reimbursing process of medical products. Additionally, future efforts should be aimed at further developing and validating the integrated modelling methodology, reaching consensus based on how to acknowledge patient heterogeneity in the economic assessment and formulating specific guidance in pharmacoeconomic guidelines. This would increase population health gains and stimulate that HTA practice and national reimbursement decisions are better aligned with the development towards personalised medicine in clinical practice.


Part of this paper was published in Bram Ramaekers’ PhD dissertation.

Abbreviations list

HRQOL, health-related quality of life; HTA, Health Technology Assessment; PBRSA, Performance-Based Risk Sharing Agreements; QALY, quality-adjusted life year.

Authors contribution

All authors contributed to the conception, design, and preparation of the manuscript, as well as read and approved the final manuscript.

Competing interests

None declared.

Conflict of interests

None declared.


All authors abide by the Association for Medical Ethics (AME) ethical rules of disclosure.


  • 1. Lahn BT, Ebenstein L. Let’s celebrate human genetic diversity. Nature 2009 Oct;461(7265):726-8.
  • 2. Chopra SS . STUDENTJAMA. Preparing for personalized medicine. JAMA 2004 Apr;291(13):1640.
  • 3. Grutters JP, Sculpher M, Briggs AH, Severens JL, Candel MJ, Stahl JE. Acknowledging patient heterogeneity in economic evaluation: a systematic literature review. Pharmacoeconomics 2013 Feb;31(2):111-23.
  • 4. Bast RC Jr, Hortobagyi GN. Individualized care for patients with cancer – a work in progress. N Engl J Med 2004 Dec;351(27):2865-7.
  • 5. Evans WE, Relling MV. Moving towards individualized medicine with pharmacogenomics. Nature 2004 May;429(6990):464-8.
  • 6. Schilsky RL . Personalized medicine in oncology: the future is now. Nat Rev Drug Discov 2010 May;9(5):363-6.
  • 7. . INAHTA. Definitions of healthcare technology and technology assessment., accessed 30 Nov 2012. .
  • 8. Jain R, Grabner M, Onukwugha E. Sensitivity analysis in cost-effectiveness studies: from guidelines to practice. Pharmacoeconomics 2011 Apr;29(4):297-314.
  • 9. Drummond MF, Sculpher MJ, Torrance GW, O’Brein BJ, Stoddart GL. Methods for the economic evaluation of health care programmes. 3rd ed. Oxford: Oxford University Press; 2005.
  • 10. O’Brien B . Economic evaluation of pharmaceuticals. Frankenstein’s monster or vampire of trials? Med Care 1996 Dec;34(Suppl 12):DS99-108.
  • 11. Briggs A, Sculpher MJ, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.
  • 12. Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making 1993 Oct–Dec;13(4):322-38.
  • 13. Chalkidou K, Walley T, Culyer A, Littlejohns P, Hoy A. Evidence-informed evidence-making. J Health Serv Res Policy 2008 Jul;13(3):167-73.
  • 14. . Council for Public Health and Health Care. Sensible and sustainable care. Den Haag: Council for Public Health and Health Care; 2006. In Dutch..
  • 15. Sculpher M . Subgroups and heterogeneity in cost-effectiveness analysis. Pharmacoeconomics 2008 Sep;26(9):799-806.
  • 16. Stevens W, Normand C. Optimisation versus certainty: understanding the issue of heterogeneity in economic evaluation. Soc Sci Med 2004 Jan;58(2):315-20.
  • 17. Coyle D, Buxton MJ, O’Brien BJ. Stratified cost-effectiveness analysis: a framework for establishing efficient limited use criteria. Health Econ 2003 May;12(5):421-7.
  • 18. Zaric GS . The impact of ignoring population heterogeneity when Markov models are used in cost-effectiveness analysis. Med Decis Making 2003 Sep–Oct;23(5):379-96.
  • 19. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess 2004 Sep;8(36):.iii–iv, ix–xi, 1-158.
  • 20. Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ 2002 Jul;11(5):415-30.
  • 21. Nixon RM, Thompson SG. Methods for incorporating covariate adjustment, subgroup analysis and between-centre differences into cost-effectiveness evaluations. Health Econ 2005 Dec;14(12):1217-29.
  • 22. Willan AR, Briggs AH, Hoch JS. Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. Health Econ 2004 May;13(5):461-75.
  • 23. Basu A, Meltzer D. Value of information on preference heterogeneity and individualized care. Med Decis Making 2007 Mar–Apr;27(2):112-27.
  • 24. Ramaekers BL, Joore MA, Grutters JP. How should we deal with patient heterogeneity in economic evaluation: a systematic review of national pharmacoeconomic guidelines. Value Health 2013 Jul–Aug;16(5):855-62.
  • 25. Saramago P . The value of further research: the added value of individual-level data. Value Health 2012 Nov;15(7):A492.
  • 26. Ramaekers BL, Pijls-Johannesma M, Joore MA, van den Ende P, Langendijk JA, Lambin P. Systematic review and meta-analysis of radiotherapy in various head and neck cancers: comparing photons, carbon-ions and protons. Cancer Treat Rev 2011 May;37(3):185-201.
  • 27. Ramaekers BL, Grutters JP, Pijls-Johannesma M, Lambin P, Joore MA, Langendijk JA. Protons in head-and-neck cancer: bridging the gap of evidence. Int J Radiat Oncol Biol Phys 2013 Apr;85(5):1282-8.
  • 28. Brennan A, Akehurst R. Modelling in health economic evaluation. What is its place? What is its value? Pharmacoeconomics 2000 May;17(5):445-59.
  • 29. Ramaekers BL, Joore MA, Lueza B, Bonastre J, Mauguen A, Pignon JP. Cost-effectiveness of modified fractionation radiotherapy versus conventional radiotherapy for unresected non-small cell lung cancer patients. J Thorac Oncol 2013 Oct;8(10):1295-307.
  • 30. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S. Grading quality of evidence and strength of recommendations. BMJ 2004 Jun;328(7454):1490.
  • 31. Garrison LP, Towse A, Briggs A, de Pouvourville G, Grueger J. Performance-based risk-sharing arrangements-good practices for design, implementation, and evaluation: report of the ISPOR good practices for performance-based risk-sharing arrangements task force. Value Health 2013 Jul–Aug;16(5):703-19.
  • 32. Godlee F, Clarke M. Why don’t we have all the evidence on oseltamivir?. BMJ 2009 Dec;339b5351.
  • 33. . Nature Publishing Group. Availability of data and materials., accessed 15 Jan 2013. .
  • 34. Boulton G . Open your minds and share your results. Nature 2012 Jun;486(7404):441.
  • 35. Garattini S, Bertele V. Europe’s opportunity to open up drug regulation. BMJ 2010 Mar;340c1578.
  • 36. de Pouvourville G . Risk-sharing agreements for innovative drugs: a new solution to old problems?. Eur J Health Econ 2006 Sep;7(3):155-7.
  • 37. . National Institute for Health and Clinical Excellence. Gefitinib for the first line treatment of locally advanced or metastatic non-small cell lung cancer. Technology Appraisal 192; 2010.
Licensee to OAPL (UK) 2014. Creative Commons Attribution License (CC-BY)