Article Text
Abstract
Aims To independently evaluate and compare the performance of the Ocular Hypertension Treatment Study-European Glaucoma Prevention Study (OHTS-EGPS) prediction equation for estimating the 5-year risk of open-angle glaucoma (OAG) in four cohorts of adults with ocular hypertension.
Methods Data from two randomised controlled trials and two observational studies were analysed individually to assess transferability of the prediction equation between different geographical locations and settings. To make best use of the data and to avoid bias, missing predictor values were imputed using multivariate imputation by chained equations. Using the OHTS-EGPS risk prediction equation, predicted risk was calculated for each patient in each cohort. We used the c-index, calibration plot and calibration slope to evaluate predictive ability of the equation.
Results Analyses were based on 393, 298, 188 and 159 patients for the Rotterdam, Moorfields, Dunfermline, and Nottingham cohorts, respectively. The discriminative ability was good, with c-indices between 0.69 and 0.83. In calibration analyses, the risk of OAG was generally overestimated, although for the Rotterdam cohort the calibration slope was close to 1 (1.09, 95% CI 0.72 to 1.46), the ideal value when there is perfect agreement between predicted and observed risks.
Conclusions The OHTS-EGPS risk prediction equation has predictive utility, but further validation in a population-based setting is needed.
Statistics from Altmetric.com
Introduction
Glaucoma is the second leading cause of blindness worldwide and is predicted to affect more than 79.6 million people by 2020.1 Elevated intraocular pressure (IOP) is an established risk factor for the development of open-angle glaucoma (OAG), the most common form of glaucoma. Ocular hypertension (OHT) affects 3–5% of individuals aged over 40; about one million people in England.2 Most ocular hypertensives are identified during a routine ‘sight’ test, and the number affected is set to increase with an ageing population. OHT cannot be prevented but can be lowered, for example, by medication or laser. Several factors influence the decision to treat ocular hypertensives, particularly the risk of developing OAG and life expectancy. In clinical practice, there is considerable variation in the management of ocular hypertensives, but in the UK, National Institute for Health and Care Excellence guidelines that suggest treatment based on IOP, central corneal thickness (CCT) and age is followed.
Risk factors for conversion from OHT to OAG have been investigated in several longitudinal population-based studies and randomised trials.3–6 Although both increased age and elevated IOP have been consistently shown to be important factors,7 patients with normal range IOP (10–21 mm Hg) can develop glaucoma. Consequently, multivariable risk prediction models have been developed in an attempt to quantify the risk of developing OAG. A systematic review8 identified three risk prediction equations based on data from two randomised controlled trials (RCTs), the Ocular Hypertension Treatment Study (OHTS)5 and the European Glaucoma Prevention Study (EGPS).6 The full and reduced OHTS5 ,9 ,10 and the OHTS-EGPS11 ,12 risk prediction equations estimate the 5-year risk of developing OAG in individuals with OHT. The OHTS-EGPS equation was developed using the OHTS observation group and the EGPS placebo group and has superseded the OHTS equations. Although the OHTS and EGPS collaboration developed three equations—the equations differed mainly in how eye-specific predictors were used—they concluded that only the equation that used the means of right and left eyes for eye-specific predictors (referred to as the ‘means’ model) was the simplest to use and the most robust to measurement variability and error.12 They validated only the ‘means’ model. The five predictors in the OHTS-EGPS risk prediction equation are age, IOP, CCT, vertical cup-to-disc (C/D) ratio, and pattern standard deviation (PSD).
Validated risk calculators have become a useful tool in risk assessment for coronary heart disease (CHD),13 and parallels have been drawn between CHD and glaucoma; both are chronic diseases with known modifiable risk factors.14 ,15 Prediction models often do not generalise beyond the population used to derive them, and so it is widely accepted that a prediction model should not be applied in clinical practice before it has been validated in at least one other population and preferably by different investigators.16 We therefore independently evaluated the performance of the risk prediction equation of the OHTS-EGPS ‘means’ model using data from two RCTs and two UK observational cohorts and compared the performance of the equation in the four populations.
Methods
Study population
Data were available from two placebo-controlled randomised trials (Moorfields Eye Hospital, London, UK, and the Rotterdam Eye Hospital, the Netherlands) of medical treatment for participants with OHT and two observational cohorts of people with OHT—one hospital based (Queen Margaret Hospital, Dunfermline, UK) and one based on a community optometry-led monitoring scheme (Queens Medical Centre, Nottingham, UK). The four cohorts are summarised in table 1, and further details are available elsewhere.8
Statistical analysis
The four cohorts were analysed individually to enable assessment of the transferability of the OHTS-EGPS risk prediction equation between different geographical locations and settings. To make the best use of the available data and to avoid bias, missing values were imputed using multivariate imputation by chained equations.17 By replacing missing values with plausible values based on the distribution of the observed data, 10 imputed datasets were created. To avoid excluding a completely missing predictor from the risk estimation, we imputed the average value from a similar dataset.
The 5-year predicted risk of developing OAG was calculated for every patient in each dataset using average baseline values of eye-specific predictors when values were available for both eyes. The OHTS-EGPS risk prediction equation is given by
where 0.91831 is the average survival probability at 5 years and
is the regression coefficient in the Cox proportional hazards model for the ith predictor and is the log HR; is an individual's value for one of the k predictors; and to are the mean values from the OHTS-EGPS cohort for the k predictors. Four of the five predictors were transformed as follows:
where CCT is the central corneal thickness, IOP is the intraocular pressure, PSD is the pattern standard deviation and VCD is the vertical cup-to-disc ratio.
An online calculator is available at http://ohts.wustl.edu/risk/calculator.html. To simplify risk assessment, a points-based system was also developed from this model by the OHTS-EGPS group.
All patients, treated and untreated, were included in the analyses. Ideally, validation should be performed using an untreated population, but this approach would have more than halved the sample size of some cohorts. We assessed the predictive ability of the equation using measures of discrimination and calibration. We assessed discriminative ability using Harrell's c-index; a c-index of 1.0 indicates perfect discrimination while a c-index of 0.5 indicates random discrimination.18 Approximate 95% CIs were computed for the c-index by applying Rubin's rules to Jackknife standard errors obtained for the c-indices of the 10 imputed datasets. Where the upper limit of the 95% CI exceeded 1.0, the value was truncated at 1.0.
Calibration refers to the agreement between predicted and observed risk. Patients in each cohort were divided into quintiles according to their predicted risk. Within each quintile, the average predicted risk was compared with the corresponding Kaplan–Meier estimate of the observed risk. Calibration plots of average observed risk against predicted 5-year risk were used to illustrate calibration; for perfect calibration, all points will lie on the 45° line. Additionally, we used the slope of the prognostic index (linear predictor), known as the calibration slope to quantify calibration. The calibration slope should ideally be 1 when predicted risks agree completely with observed risk.19 Note that while the tests and summary statistics are based on appropriate analyses of all 10 imputed datasets, each plot depicts a single imputed dataset and thus may not directly reflect the overall results.
All analyses were performed using Stata V.11.1 (StataCorp LP, College Station, Texas, USA).
Results
Data completeness
All cohorts provided complete data on age. The proportion of missing values for CCT was high (between 23% and 100% missing) in all cohorts. CCT was sporadically collected for both the Moorfields and Dunfermline cohorts. However, in the Dunfermline cohort, CCT was not recorded in the electronic patient record system and so we imputed the average value (556 µm) from the Nottingham cohort to enable calculation of each patient's risk. Corrected PSD was recorded more frequently than PSD for the Moorfields cohort, which accounts for the large proportion of missing values (155 out of 298, 52%) for PSD. The number of complete cases was 238 (61%), 78 (26%), 135 (72%) and 112 (70%) for the Rotterdam, Moorfields, Dunfermline and Nottingham cohorts, respectively.
Validation sample
The median follow-up times of our observational cohorts (2.7 years for Dunfermline and 4.3 years for Nottingham) were much shorter than those of our trial-based cohorts (8.2 years for Rotterdam and 9.3 years for Moorfields). The median follow-up time was 4.8 and 6.6 years for the EGPS and the OHTS, respectively. The Kaplan–Meier estimate of the 5-year cumulative probability of developing OAG was 9.3% in the OHTS observation group and 16.8% in the EGPS placebo group. These values were 4.0% (95% CI 2.3% to 6.9%) for Rotterdam, 11.3% (95% CI 7.8% to 16.4%) for Moorfields, 23.7% (95% CI 14.1% to 39.9%) for Dunfermline and 5.1% (95% CI 2.0% to 12.7%) for Nottingham.
The baseline characteristics of our four cohorts were generally similar to the OHTS-EGPS cohort, although patients in our observational cohorts were, on average, older than those in the randomised cohorts (table 2). On average, patients in the Moorfields cohort had a thinner CCT compared with those in the OHTS-EGPS, Rotterdam or Nottingham cohorts. The average PSD in the group that converted to OAG and the group that did not was lower in the Rotterdam cohort compared with the respective groups in other cohorts. Examination of the distribution of imputed values showed similarity with observed values, and the imputed data were used for analyses based on 393, 298, 188 and 159 patients for the Rotterdam, Moorfields, Dunfermline and Nottingham cohorts, respectively.
Model performance
The c-indices were between 0.69 and 0.83 (table 3); the OHTS-EGPS risk prediction equation showed the best discriminatory performance in the Rotterdam cohort (0.83, 95% CI 0.75 to 0.91) but the worst performance in the Moorfields cohort (0.69, 95% CI 0.59 to 0.78). The calibration of the model in the four cohorts is illustrated in figure 1. The plots should be viewed cautiously because they were based on small study sizes and few conversions to OAG in each quintile, and they depict data from only one imputed dataset per cohort, which may be atypical. In contrast, the calibration slopes presented in table 3 were computed based on the 10 imputed datasets. Unlike the other cohorts where the calibration slopes were less than 1, the calibration slope for the Rotterdam cohort was close to 1 (1.09, 95% CI 0.72 to 1.46), the ideal value when there is complete agreement between predicted and observed risks.
Discussion
We independently assessed the performance of the OHTS-EGPS equation for predicting the 5-year risk of OAG in cohorts of patients from primary and secondary care. The ability of the equation to distinguish between ocular hypertensives who developed OAG and those who did not in the four cohorts was good, with c-indices ranging between 0.69 and 0.83. The c-indices were similar to those of the Framingham model for CHD (between 0.69 and 0.83 in six different cohorts)20 and better than those of the Gail model for invasive breast cancer (0.58, 95% CI 0.56 to 0.60).21 A c-index of 0.83 indicates that, in approximately 83% of the cases, the model allocated a higher predicted probability for an individual who converted than for one who did not.
In calibration analyses, the OHTS-EGPS equation generally overestimated the risk of OAG. The calibration slope in the Rotterdam cohort (1.09) was similar to that of the OHTS equation (1.086) in the Diagnostic Innovations in Glaucoma Study (DIGS) cohort.10 In both cases, the values were not statistically different from 1, the ideal value. For our other cohorts, the calibration slopes were much lower than one.
Variation in the performance of the OHTS-EGPS equation between cohorts may be due to differences in patient characteristics and inconsistency in the definition of glaucoma. Our RCTs used patient selection criteria that were similar to those of the OHTS and EGPS. In contrast, the criteria used in our observational cohorts were more flexible. Glaucoma is diagnosed on the basis of a characteristic pattern of damage to the optic nerve with a corresponding and reproducible visual field defect, but there is no ‘gold’ standard case definition of early glaucoma. The OHTS and EGPS allowed conversion based on optic nerve criteria alone without the presence of a glaucomatous visual field defect. This was less stringent than in our trial cohorts where conversion to glaucoma required the presence of a visual field defect. However, the conversion criteria used in our observational cohorts were not as rigid and well defined (see table 1). The latter is likely to be a better reflection of clinical practice than the criteria and processes adopted in the trials. A risk prediction equation is potentially most valuable in this context as an aid to define criteria for monitoring and treatment guidelines for the management of OHT. Since the discriminatory ability was generally good, the equation can be adjusted or recalibrated to correct for optimism. However, given the limitations of our data (large proportions of missing data in all cohorts and in some cohorts there were few events relative to the number of predictors in the equation), we were cautious and did not attempt recalibration because the validity of such analyses may be questionable.
The OHTS-EGPS risk prediction equation is a practical tool because all the variables can be routinely collected in clinical practice. However, it does not include other major risk factors such as race and family history of glaucoma, though it has been suggested that black race may not be an independent risk factor. Black patients tend to have higher IOP, thinner corneas and greater cup-to-disc ratios than other patients with OHT and are therefore generally at a higher risk than white patients.22 Other ocular conditions such as pseudoexfoliation (PEX)3 ,6 ,23 and optic disc haemorrhage24 ,25 have been found to be predictive for the development of OAG. The OHTS excluded individuals with PEX and pigment dispersion while the EGPS had only 19 individuals with these conditions. They were excluded in the pooled OHTS-EGPS analyses. Glaucoma can affect either or both eyes, but it is often asymmetric occurring first in one eye only. The equation uses the means of both eyes, thus including relevant data from both eyes of each patient but potentially at the expense of predictive accuracy.
Limitations of the study
One of the limitations of this study was the inclusion of treated patients in all cohorts, though the only predictor affected by treatment is IOP. Since baseline IOP values were used in the prediction equation, the effect of treatment may be important for those who were untreated at baseline but less important for those who were already on treatment at baseline as may be the case with the observational (Dunfermline and Nottingham) cohorts. Although the OHTS-EGPS model was developed to estimate the risk for a patient left untreated for 5 years, the HRs were similar to those of the OHTS model based on treated and untreated patients. This OHTS model was validated in an untreated population and similar HRs were obtained for the predictors.10 We investigated the effect of treatment by performing Cox regression analyses with and without stratification by treatment on each imputed dataset so that we could compare HRs. The results obtained were similar. In sensitivity analyses, for the Rotterdam and Moorfields cohorts, we also produced calibration plots separately for the treated, untreated and combined data. The effect of IOP-lowering treatment on calibration was unclear from the plots.
A second limitation was the number of OAG cases in our cohorts. Although all cohorts were larger than the DIGS cohort used for validation of the OHTS equations, OAG cases were fewer except in the Moorfields cohort. The Dunfermline (secondary care) and Nottingham (community care) cohorts may be unrepresentative of the wider OHT population because in the UK patients considered to have a moderate or high risk are often kept in secondary care while ocular hypertensives deemed to be very low risk are monitored in primary care (optometry). In addition to shorter follow-up, this may explain the lower incidence of OAG in the Dunfermline (28/188, 14.9%) and Nottingham (5/159, 3.1%) cohorts compared with the DIGS cohort (31/126, 24.6%).
A third limitation was the large proportion of missing values for some predictors, especially CCT. CCT was not routinely collected prior to the publication of the findings of the OHTS study. The OHTS was the first study to document that a thinner central corneal measurement predicts the development of OAG and recommended its measurement in the clinical evaluation of patients with OHT.5
In conclusion, to guide therapeutic intervention and for the prevention of OAG, there is a need for accurate and reliable risk assessment for conversion from OHT to OAG. Despite the limitations of our cohorts, the OHTS-EGPS risk prediction equation generally discriminated well in estimating the 5-year risk of OAG but was not well calibrated in all populations. Therefore, further evaluation in a population-based setting is needed. The equation has not been revised since its original publication, and future research may seek to update the equation to reflect the role and impact of other risk factors in order to improve predictive ability.
Acknowledgments
The authors would like to thank the following for their assistance in the collation of the data for the validation datasets used in this study: Josine van der Schoot and the Rotterdam Ophthalmic Institute (Rotterdam, the Netherlands); Ryo Asaoka (Moorfields Eye Hospital, UK); Karen Armstrong Owen (Queens Medical Centre, Nottingham, UK); and David Holgate and Stuart Dobson (Queen Margaret Hospital, Dunfermline, UK).
References
Footnotes
-
Collaborators Surveillance for Ocular Hypertension Study Group: Jennifer Burr, Adriana Botello, Yemisi Takwoingi, Rodolfo Hernández, Maria Vazquez-Montes, Andrew Elders, Ryo Asaoka, Katie Banister, Josine van der Schoot, Cynthia Fraser, Anthony King, Hans Lemij, Roshini Sanders, Stephen Vernon, Anja Tuulonen, Aachal Kotecha, Paul Glasziou, David Garway-Heath, David Crabb, Luke Vale, Augusto Azuara-Blanco, Rafael Perera, Mandy Ryan, Jonathan Deeks and Jonathan Cook.
-
Contributors JMB, RS and JJD contributed to the conception and design of the study. JMB, AA-B, HGL and JJD obtained funding. JMB, APB, DFG-H, HGL, RS and AJK contributed to data acquisition. YT, APB, JMB, AA-B, HGL and JJD contributed to data analysis. YT conducted statistical analysis under supervision by JJD and drafted the article. YT, APB and AJK provided administrative, technical and material support. All authors contributed to the interpretation of the results, revised the article critically for important intellectual content and gave final approval for publication.
-
Funding This work was part of the Surveillance for Ocular Hypertension study funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme (07/46/02). The funding organisation had no role in the design or conduct of this research.
-
Competing interests DFG-H acknowledges a proportion of his financial support from the Department of Health through the award made by the NIHR to the Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology. DFG-H’s chair at UCL is supported by funding from the International Glaucoma Association.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement All authors agree to allow review of the data by British Journal of Ophthalmology upon request from the corresponding author.