Aim: To assess the validity of the time trade-off (TTO) and standard reference gamble (SRG) techniques of utility assessment in patients with retinal disease. A cross section of eligible patients was studied and validity was determined through their relation with two logical constructs, visual acuity and scores from the Visual Function 14 (VF-14) index.
Methods: The study consisted of eligible patients presenting to a tertiary retinal facility who completed an interview. All patients had best corrected vision of 20/40 or worse in at least one eye. TTO and SRG utilities, as well as a VF-14 questionnaire, were administered through a standardised interview. Demographic and clinical (including Snellen visual acuity) information was also collected.
Results: 323 patients met these study criteria. Significant predictors of TTO utilities in the multivariate analysis were vision in the better seeing eye (p<0.01) and VF-14 scores (p<0.01). Significant predictors of standard gamble utilities were also vision in the better seeing eye (p<0.01) and VF-14 scores (p<0.05).
Conclusion: Both the standard gamble and TTO methods demonstrate strong validity when evaluated against visual acuity in the better seeing eye and the VF-14 score.
- time trade-off
- standard gamble
Statistics from Altmetric.com
Utility instruments measure health related quality of life (HRQL) and produce a single number representing the value or preference that a person attaches to a particular health state.1 Consequently, utility valuation allows for the objective measurement of the desirability of a particular disease state according to a patient's own perception of their life.2 Many utility valuation instruments have been introduced and applied including the standard reference gamble (SRG) first introduced by Von Neumann and Morgenstern,3 and the time trade-off (TTO) method introduced by Torrance et al.4
HRQL can also be measured using generic questionnaires, which attempt to measure overall HRQL, or specific questionnaires, which focus on certain aspects of health such as disease, population, or function. Specific HRQL questionnaires give more information about specific aspects of health (for example, visual function) and are considered more responsive to changing health states, but may be limited in terms of their generalisability to different populations or interventions.5 In ophthalmology, specific HRQL questionnaires include the 51 item National Eye Institute visual function questionnaire6 and the Visual Function 14 (VF-14) questionnaire,7 which was first used for cataract patients. The VF-14 has been since validated for patients with retinal disease.8
Although there is no accepted gold standard in utility assessment, qualities that are required in a good utility assessment tool have been described. According to Guyatt et al,9 it is imperative that a utility assessment tool possess construct validity, as well as being reliable, easy to administer, responsive, and interpretable. Measuring the construct validity of a utility instrument requires comparing the utility values with other patient characteristics and examining logical relations, which should exist.5,9 Consequently, a utility measure should correlate well with clinical measures of disease and disability, as well as with specific HRQL questionnaires, which relate to the disease in question. It is especially critical that the validity of these utility valuation techniques be tested, considering that they are being used as the backbone of a number of cost effectiveness analyses for treatments of ocular disease.10–,14
It has been demonstrated that visual acuity in the better seeing eye is directly related to TTO and SRG utility scores in patients with retinal disease.2,14 It is the purpose of this study to investigate the construct validity of the TTO and SRG techniques of utility assessment by assessing their relation to two logical constructs simultaneously—visual acuity and the VF-14 HRQL questionnaire.
This study employed a cross sectional design. Patients were interviewed by one of the authors (GCB) at a retina clinic in a tertiary hospital in Philadelphia, PA, USA. Patients were eligible for the study if they had 20/40 vision or worse in at least one eye and were deemed competent to answer the required questions. Patients were excluded for communication barriers, developmental disability, and psychiatric illness. A thorough ocular examination was initially performed which included Snellen visual acuity and improvements using a pinhole. Vision in the better seeing eye, vision in the worse seeing eye, and primary reason for visual loss were recorded.
A standardised interview was performed by an experienced researcher trained in utility valuation. Demographic and clinical data including age, sex, ethnicity, years of formal education, employment status, number of co-morbid diseases, and duration of visual loss were collected through the interview. Finally, TTO and SRG utility valuation was undertaken. In addition, the VF-14 questionnaire was administered. The methodology pertaining to the evaluation of visual utilities is as follows:
Time trade-off utilities
TTO visual utility values were determined by responses to two hypothetical questions. Firstly, patients were asked to estimate their remaining life expectancy. Next, patients were told to consider a hypothetical situation where a technology existed that could permanently return their vision to normal, would always work, but would decrease their survival. Patients were then asked to quantify the maximum number of years out of their expected life expectancy, if any, that they would be willing to trade in return for normal vision. These two pieces of information are used to determine the utility value for the patient as follows:
For example, if a patient expects to live 16 years and would be willing to give up 4 years for perfect vision, then the visual utility is calculated as (16 years − 4 years)/16 years = 0.75. The visual utility is a value between 0 (defined as willingness to trade off all remaining life for perfect vision), and 1.0 (defined as not willing to trade off any years).
Standard reference gamble utilities
SRG utilities were ascertained through answers to a similar question about a hypothetical technology. Patients were asked to consider a scenario where a new technology for their eye problem exists. When this technology works, they would receive perfect vision in both eyes for the rest of their lives. However, when the procedure fails, they would not survive. Patient were asked to estimate the largest percentage risk of death they would be willing to accept to be relieved of their ocular disease. The SRG utility is simply this percentage. Consequently, if a patient perceives an 80% chance of having a successful treatment (that is, returning eyesight to normal) and a 20% chance of death as being equivalent to maintaining their current visual state, then his or her SRG visual utility is 0.8.
Data management and analysis
Data were entered, managed, and analysed using spss 10.0 for Windows. Demographic characteristics, clinical characteristics, visual utility values (using both the SRG and TTO methods), and VF-14 scores were displayed for the sample. The main outcome variable of interest was visual utility scores (using TTO and SRG techniques). Bivariate analyses were performed to determine the association between visual utility and the variables of interest (demographic characteristics, clinical characteristics, and VF-14 scores). Pearson correlation coefficients and analysis of variance (ANOVA) were used with appropriate significance tests. Also, a cross tabulation between visual acuity in the better seeing eye with VF-14 scores, TTO, and SRG utility scores was displayed.
In order to investigate the construct validity of the two utility assessment techniques, backward multiple linear regression, with an exclusion of p=0.05, was employed. Two regression analyses were performed, one using TTO utility scores as the outcome variable in order to assess the validity of the TTO, and the other using SRG utility scores as the outcome variable, in order to assess the validity of the SRG. All available independent variables (visual acuities, VF-14 scores, demographic variables, and clinical variables) were used as potential predicting variables.
Description of sample
Three hundred and twenty three eligible patients completed the entire survey. Demographic characteristics of the sample are shown in Table 1⇓. The mean age of the sample was 67.5 years (SD 11.9 years), 63.5% were female, and over 96% were white. Over 40% of the sample had some formal post-secondary education, and over 50% of the sample was retired while just under 40% were employed.
The clinical characteristics of the sample are shown in Table 2⇓. Over 60% of the sample had visual loss in the affected eye of 6/60 or worse. The median Snellen acuity in the unaffected eye was 6/12; whereas, the median acuity in the affected eye was 6/90. Patients suffered from their ocular condition for an average of 2.9 years (SD 5.03 years). One hundred and seven patients (33.1%) suffered from age related macular degeneration, 105 (32.5%) suffered from diabetic retinopathy, and 111 (34.4%) suffered from other ocular diseases including cataract, glaucoma, retinal detachment, non-diabetic oedema, amblyopia, vascular obstruction, and corneal disease.
Visual utility and VF-14 responses are shown in Table 3⇓. Patients expected to live, on average, 15.3 years (SD, 9.4 years) from their present age. The mean TTO visual utility was 0.770 (SD 0.228) while the mean SRG visual utility was 0.853 (SD 0.21). The mean VF-14 score, out of 100, was 68.46 (SD 28.43). Visual utility scores (TTO and SRG), and VF-14 scores are shown stratified by vision in the better seeing eye in Table 4⇓. As vision in the better seeing eye worsened, the VF-14 scores, TTO visual utilities, and SRG visual utilities decreased significantly (p<0.0005 for each variable).
Bivariate analyses were performed in order to determine which variables were independently associated with visual utility scores. TTO utility scores were not significantly associated with any of the demographic variables including age, sex, ethnicity, years of formal education, or employment status at the 5% level. In addition, TTO utility scores were not significantly associated with duration of visual loss at the 5% level. However, TTO utility scores were significantly associated with visual acuity in the affected (p<0.01) and unaffected (p<0.01) eye, reason for visual loss (p<0.01), and number of co-morbid diseases (r = −0.152, p=0.006). Better visual acuity in the affected and unaffected eye was both independently associated with higher utility scores (p<0.01).
Bivariate associations between SRG visual utilities and the variables of interest were similar to the associations with TTO utilities. SRG utilities were not significantly associated with any of the demographic characteristics available at the 5% level. Better visual acuity in both the affected (p<0.01) and unaffected (p<0.01) eye was independently associated with higher utility scores.
Separate multivariate analyses were performed for both TTO utilities and SRG utilities as dependent variables, respectively, and are shown in Tables 5 and 6⇓⇓. Visual acuity in the better seeing eye (p<0.01), and VF-14 scores (p<0.01) were the only variables that significantly predicted TTO utilities. The model was highly significant (F=61.9, p<0.01).
The results of the backward regression analysis using SRG visual utilities, as the dependent variable, were very similar to the results for TTO utilities. Again, visual acuity in the better seeing eye (p<0.01), and VF-14 scores (p<0.05) were the only variables that significantly predicted SRG utility scores. This model was also highly significant (F=36.1, p<0.01).
Utilities are currently being used in cost utility analyses that investigate the cost effectiveness of various treatments for ocular diseases.10–,13 It is important that the techniques being employed have significant construct validity. This is the first study, to our knowledge, to examine the relation between utilities and both visual acuity and VF-14 scores, as well as other potential covariates, simultaneously. By examining this relation we can assess the construct validity of the SRG and TTO utility methods using visual acuity and VF-14 scores as the logical constructs.
Our results demonstrate that both the SRG and TTO techniques show good construct validity, as they are highly associated with both visual acuity in the better seeing eye and scores on the VF-14 questionnaire simultaneously. In addition, none of the other demographic or clinical variables was significantly associated with the utility values when visual acuity and VF-14 scores were accounted for.
Two limitations with this study must be noted. Firstly, the study used a very specific group of older patients with ocular diseases of the retina. Therefore, the generalisability of these results to other groups of patients is not known. Secondly, we have only used a visual specific HRQL questionnaire and did not employ a general HRQL questionnaire such as the SF-36. Therefore, perhaps a general HRQL questionnaire would allow us to explain more of the variation in utility values, which was observed. We are currently performing a study that will allow us to evaluate the validity of utility valuation as measured against the construct of the SF-36 score.
Financial support: The Canadian Foundation for Innovation, Ottawa, Ontario; The JP Bickell Foundation, Toronto, Ontario; Jeanne Mance Foundation, Hotel Dieu Hospital, Kingston, Ontario; and the EA Baker Foundation, Canadian National Institute for the Blind, Toronto Ontario, Canada.
Dr Sharma holds a New Investigator Award from the Canadian Institutes for Health Research, Ottawa, Ontario and is a recipient of the Ontario Ministry of Energy, Science and Technology's Premier's Research Excellence Awards, Toronto, Ontario.
Series editors: G and M Brown