Article Text

Download PDFPDF

External validation of the OHTS-EGPS model for predicting the 5-year risk of open-angle glaucoma in ocular hypertensives
  1. Yemisi Takwoingi1,
  2. Adriana P Botello2,
  3. Jennifer M Burr3,
  4. Augusto Azuara-Blanco4,
  5. David F Garway-Heath5,
  6. Hans G Lemij6,
  7. Roshini Sanders7,
  8. Anthony J King8,
  9. Jonathan J Deeks1,
  10. for the Surveillance for Ocular Hypertension Study Group
  1. 1Department of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham, UK
  2. 2Aberdeen Royal Infirmary, Aberdeen, UK
  3. 3School of Medicine, University of St Andrews, St Andrews, UK
  4. 4Centre for Vision and Vascular Science, Queen's University Belfast, Belfast, UK
  5. 5NIHR Biomedical Research Centre at Moorfields Eye Hospital and UCL Institute of Ophthalmology, London, UK
  6. 6Glaucoma Services, Rotterdam Eye Hospital, Rotterdam, The Netherlands
  7. 7Queen Margaret Hospital, Dunfermline, UK
  8. 8Department of Ophthalmology, Queen's Medical Centre, University Hospital, Nottingham, UK
  1. Correspondence to Dr Yemisi Takwoingi, Department of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham B15 2TT, UK; y.takwoingi{at}bham.ac.uk

Abstract

Aims To independently evaluate and compare the performance of the Ocular Hypertension Treatment Study-European Glaucoma Prevention Study (OHTS-EGPS) prediction equation for estimating the 5-year risk of open-angle glaucoma (OAG) in four cohorts of adults with ocular hypertension.

Methods Data from two randomised controlled trials and two observational studies were analysed individually to assess transferability of the prediction equation between different geographical locations and settings. To make best use of the data and to avoid bias, missing predictor values were imputed using multivariate imputation by chained equations. Using the OHTS-EGPS risk prediction equation, predicted risk was calculated for each patient in each cohort. We used the c-index, calibration plot and calibration slope to evaluate predictive ability of the equation.

Results Analyses were based on 393, 298, 188 and 159 patients for the Rotterdam, Moorfields, Dunfermline, and Nottingham cohorts, respectively. The discriminative ability was good, with c-indices between 0.69 and 0.83. In calibration analyses, the risk of OAG was generally overestimated, although for the Rotterdam cohort the calibration slope was close to 1 (1.09, 95% CI 0.72 to 1.46), the ideal value when there is perfect agreement between predicted and observed risks.

Conclusions The OHTS-EGPS risk prediction equation has predictive utility, but further validation in a population-based setting is needed.

View Full Text

Statistics from Altmetric.com

Introduction

Glaucoma is the second leading cause of blindness worldwide and is predicted to affect more than 79.6 million people by 2020.1 Elevated intraocular pressure (IOP) is an established risk factor for the development of open-angle glaucoma (OAG), the most common form of glaucoma. Ocular hypertension (OHT) affects 3–5% of individuals aged over 40; about one million people in England.2 Most ocular hypertensives are identified during a routine ‘sight’ test, and the number affected is set to increase with an ageing population. OHT cannot be prevented but can be lowered, for example, by medication or laser. Several factors influence the decision to treat ocular hypertensives, particularly the risk of developing OAG and life expectancy. In clinical practice, there is considerable variation in the management of ocular hypertensives, but in the UK, National Institute for Health and Care Excellence guidelines that suggest treatment based on IOP, central corneal thickness (CCT) and age is followed.

Risk factors for conversion from OHT to OAG have been investigated in several longitudinal population-based studies and randomised trials.3–6 Although both increased age and elevated IOP have been consistently shown to be important factors,7 patients with normal range IOP (10–21 mm Hg) can develop glaucoma. Consequently, multivariable risk prediction models have been developed in an attempt to quantify the risk of developing OAG. A systematic review8 identified three risk prediction equations based on data from two randomised controlled trials (RCTs), the Ocular Hypertension Treatment Study (OHTS)5 and the European Glaucoma Prevention Study (EGPS).6 The full and reduced OHTS5 ,9 ,10 and the OHTS-EGPS11 ,12 risk prediction equations estimate the 5-year risk of developing OAG in individuals with OHT. The OHTS-EGPS equation was developed using the OHTS observation group and the EGPS placebo group and has superseded the OHTS equations. Although the OHTS and EGPS collaboration developed three equations—the equations differed mainly in how eye-specific predictors were used—they concluded that only the equation that used the means of right and left eyes for eye-specific predictors (referred to as the ‘means’ model) was the simplest to use and the most robust to measurement variability and error.12 They validated only the ‘means’ model. The five predictors in the OHTS-EGPS risk prediction equation are age, IOP, CCT, vertical cup-to-disc (C/D) ratio, and pattern standard deviation (PSD).

Validated risk calculators have become a useful tool in risk assessment for coronary heart disease (CHD),13 and parallels have been drawn between CHD and glaucoma; both are chronic diseases with known modifiable risk factors.14 ,15 Prediction models often do not generalise beyond the population used to derive them, and so it is widely accepted that a prediction model should not be applied in clinical practice before it has been validated in at least one other population and preferably by different investigators.16 We therefore independently evaluated the performance of the risk prediction equation of the OHTS-EGPS ‘means’ model using data from two RCTs and two UK observational cohorts and compared the performance of the equation in the four populations.

Methods

Study population

Data were available from two placebo-controlled randomised trials (Moorfields Eye Hospital, London, UK, and the Rotterdam Eye Hospital, the Netherlands) of medical treatment for participants with OHT and two observational cohorts of people with OHT—one hospital based (Queen Margaret Hospital, Dunfermline, UK) and one based on a community optometry-led monitoring scheme (Queens Medical Centre, Nottingham, UK). The four cohorts are summarised in table 1, and further details are available elsewhere.8

Table 1

Summary of the validation cohorts

Statistical analysis

The four cohorts were analysed individually to enable assessment of the transferability of the OHTS-EGPS risk prediction equation between different geographical locations and settings. To make the best use of the available data and to avoid bias, missing values were imputed using multivariate imputation by chained equations.17 By replacing missing values with plausible values based on the distribution of the observed data, 10 imputed datasets were created. To avoid excluding a completely missing predictor from the risk estimation, we imputed the average value from a similar dataset.

The 5-year predicted risk of developing OAG was calculated for every patient in each dataset using average baseline values of eye-specific predictors when values were available for both eyes. The OHTS-EGPS risk prediction equation is given byEmbedded Image

where 0.91831 is the average survival probability at 5 years andEmbedded Image

Embedded Image is the regression coefficient in the Cox proportional hazards model for the ith predictor and is the log HR; Embedded Image is an individual's value for one of the k predictors; and Embedded Image to Embedded Image are the mean values from the OHTS-EGPS cohort for the k predictors. Four of the five predictors were transformed as follows:Embedded Image Embedded Image Embedded Image Embedded Image

where CCT is the central corneal thickness, IOP  is the intraocular pressure, PSD is the pattern standard deviation and VCD is the vertical cup-to-disc ratio.

An online calculator is available at http://ohts.wustl.edu/risk/calculator.html. To simplify risk assessment, a points-based system was also developed from this model by the OHTS-EGPS group.

All patients, treated and untreated, were included in the analyses. Ideally, validation should be performed using an untreated population, but this approach would have more than halved the sample size of some cohorts. We assessed the predictive ability of the equation using measures of discrimination and calibration. We assessed discriminative ability using Harrell's c-index; a c-index of 1.0 indicates perfect discrimination while a c-index of 0.5 indicates random discrimination.18 Approximate 95% CIs were computed for the c-index by applying Rubin's rules to Jackknife standard errors obtained for the c-indices of the 10 imputed datasets. Where the upper limit of the 95% CI exceeded 1.0, the value was truncated at 1.0.

Calibration refers to the agreement between predicted and observed risk. Patients in each cohort were divided into quintiles according to their predicted risk. Within each quintile, the average predicted risk was compared with the corresponding Kaplan–Meier estimate of the observed risk. Calibration plots of average observed risk against predicted 5-year risk were used to illustrate calibration; for perfect calibration, all points will lie on the 45° line. Additionally, we used the slope of the prognostic index (linear predictor), known as the calibration slope to quantify calibration. The calibration slope should ideally be 1 when predicted risks agree completely with observed risk.19 Note that while the tests and summary statistics are based on appropriate analyses of all 10 imputed datasets, each plot depicts a single imputed dataset and thus may not directly reflect the overall results.

All analyses were performed using Stata V.11.1 (StataCorp LP, College Station, Texas, USA).

Results

Data completeness

All cohorts provided complete data on age. The proportion of missing values for CCT was high (between 23% and 100% missing) in all cohorts. CCT was sporadically collected for both the Moorfields and Dunfermline cohorts. However, in the Dunfermline cohort, CCT was not recorded in the electronic patient record system and so we imputed the average value (556 µm) from the Nottingham cohort to enable calculation of each patient's risk. Corrected PSD was recorded more frequently than PSD for the Moorfields cohort, which accounts for the large proportion of missing values (155 out of 298, 52%) for PSD. The number of complete cases was 238 (61%), 78 (26%), 135 (72%) and 112 (70%) for the Rotterdam, Moorfields, Dunfermline and Nottingham cohorts, respectively.

Validation sample

The median follow-up times of our observational cohorts (2.7 years for Dunfermline and 4.3 years for Nottingham) were much shorter than those of our trial-based cohorts (8.2 years for Rotterdam and 9.3 years for Moorfields). The median follow-up time was 4.8 and 6.6 years for the EGPS and the OHTS, respectively. The Kaplan–Meier estimate of the 5-year cumulative probability of developing OAG was 9.3% in the OHTS observation group and 16.8% in the EGPS placebo group. These values were 4.0% (95% CI 2.3% to 6.9%) for Rotterdam, 11.3% (95% CI 7.8% to 16.4%) for Moorfields, 23.7% (95% CI 14.1% to 39.9%) for Dunfermline and 5.1% (95% CI 2.0% to 12.7%) for Nottingham.

The baseline characteristics of our four cohorts were generally similar to the OHTS-EGPS cohort, although patients in our observational cohorts were, on average, older than those in the randomised cohorts (table 2). On average, patients in the Moorfields cohort had a thinner CCT compared with those in the OHTS-EGPS, Rotterdam or Nottingham cohorts. The average PSD in the group that converted to OAG and the group that did not was lower in the Rotterdam cohort compared with the respective groups in other cohorts. Examination of the distribution of imputed values showed similarity with observed values, and the imputed data were used for analyses based on 393, 298, 188 and 159 patients for the Rotterdam, Moorfields, Dunfermline and Nottingham cohorts, respectively.

Table 2

Baseline clinical and demographic characteristics for the OHTS-EGPS derivation cohort and the four validation cohorts

Model performance

The c-indices were between 0.69 and 0.83 (table 3); the OHTS-EGPS risk prediction equation showed the best discriminatory performance in the Rotterdam cohort (0.83, 95% CI 0.75 to 0.91) but the worst performance in the Moorfields cohort (0.69, 95% CI 0.59 to 0.78). The calibration of the model in the four cohorts is illustrated in figure 1. The plots should be viewed cautiously because they were based on small study sizes and few conversions to OAG in each quintile, and they depict data from only one imputed dataset per cohort, which may be atypical. In contrast, the calibration slopes presented in table 3 were computed based on the 10 imputed datasets. Unlike the other cohorts where the calibration slopes were less than 1, the calibration slope for the Rotterdam cohort was close to 1 (1.09, 95% CI 0.72 to 1.46), the ideal value when there is complete agreement between predicted and observed risks.

Table 3

Performance statistics for the OHTS-EGPS equation for predicting the 5-year risk of glaucoma in the four validation cohorts

Figure 1

Calibration plot for each validation cohort. For each quintile, the average observed risk is plotted with its 95% CI against the predicted 5-year risk of developing open-angle glaucoma (OAG). The dotted straight line through the origin represents perfect calibration. The histogram above the horizontal axis represents the distribution of predicted risks. The plots should be viewed cautiously because they were based on small study sizes and few conversions to OAG, and they depict only a single imputed dataset, which may be atypical unlike the calibration slopes presented in table 3 that were based on all 10 imputed datasets for each cohort. Reproduced with permission from Burr J, et al.8

Discussion

We independently assessed the performance of the OHTS-EGPS equation for predicting the 5-year risk of OAG in cohorts of patients from primary and secondary care. The ability of the equation to distinguish between ocular hypertensives who developed OAG and those who did not in the four cohorts was good, with c-indices ranging between 0.69 and 0.83. The c-indices were similar to those of the Framingham model for CHD (between 0.69 and 0.83 in six different cohorts)20 and better than those of the Gail model for invasive breast cancer (0.58, 95% CI 0.56 to 0.60).21 A c-index of 0.83 indicates that, in approximately 83% of the cases, the model allocated a higher predicted probability for an individual who converted than for one who did not.

In calibration analyses, the OHTS-EGPS equation generally overestimated the risk of OAG. The calibration slope in the Rotterdam cohort (1.09) was similar to that of the OHTS equation (1.086) in the Diagnostic Innovations in Glaucoma Study (DIGS) cohort.10 In both cases, the values were not statistically different from 1, the ideal value. For our other cohorts, the calibration slopes were much lower than one.

Variation in the performance of the OHTS-EGPS equation between cohorts may be due to differences in patient characteristics and inconsistency in the definition of glaucoma. Our RCTs used patient selection criteria that were similar to those of the OHTS and EGPS. In contrast, the criteria used in our observational cohorts were more flexible. Glaucoma is diagnosed on the basis of a characteristic pattern of damage to the optic nerve with a corresponding and reproducible visual field defect, but there is no ‘gold’ standard case definition of early glaucoma. The OHTS and EGPS allowed conversion based on optic nerve criteria alone without the presence of a glaucomatous visual field defect. This was less stringent than in our trial cohorts where conversion to glaucoma required the presence of a visual field defect. However, the conversion criteria used in our observational cohorts were not as rigid and well defined (see table 1). The latter is likely to be a better reflection of clinical practice than the criteria and processes adopted in the trials. A risk prediction equation is potentially most valuable in this context as an aid to define criteria for monitoring and treatment guidelines for the management of OHT. Since the discriminatory ability was generally good, the equation can be adjusted or recalibrated to correct for optimism. However, given the limitations of our data (large proportions of missing data in all cohorts and in some cohorts there were few events relative to the number of predictors in the equation), we were cautious and did not attempt recalibration because the validity of such analyses may be questionable.

The OHTS-EGPS risk prediction equation is a practical tool because all the variables can be routinely collected in clinical practice. However, it does not include other major risk factors such as race and family history of glaucoma, though it has been suggested that black race may not be an independent risk factor. Black patients tend to have higher IOP, thinner corneas and greater cup-to-disc ratios than other patients with OHT and are therefore generally at a higher risk than white patients.22 Other ocular conditions such as pseudoexfoliation (PEX)3 ,6 ,23 and optic disc haemorrhage24 ,25 have been found to be predictive for the development of OAG. The OHTS excluded individuals with PEX and pigment dispersion while the EGPS had only 19 individuals with these conditions. They were excluded in the pooled OHTS-EGPS analyses. Glaucoma can affect either or both eyes, but it is often asymmetric occurring first in one eye only. The equation uses the means of both eyes, thus including relevant data from both eyes of each patient but potentially at the expense of predictive accuracy.

Limitations of the study

One of the limitations of this study was the inclusion of treated patients in all cohorts, though the only predictor affected by treatment is IOP. Since baseline IOP values were used in the prediction equation, the effect of treatment may be important for those who were untreated at baseline but less important for those who were already on treatment at baseline as may be the case with the observational (Dunfermline and Nottingham) cohorts. Although the OHTS-EGPS model was developed to estimate the risk for a patient left untreated for 5 years, the HRs were similar to those of the OHTS model based on treated and untreated patients. This OHTS model was validated in an untreated population and similar HRs were obtained for the predictors.10 We investigated the effect of treatment by performing Cox regression analyses with and without stratification by treatment on each imputed dataset so that we could compare HRs. The results obtained were similar. In sensitivity analyses, for the Rotterdam and Moorfields cohorts, we also produced calibration plots separately for the treated, untreated and combined data. The effect of IOP-lowering treatment on calibration was unclear from the plots.

A second limitation was the number of OAG cases in our cohorts. Although all cohorts were larger than the DIGS cohort used for validation of the OHTS equations, OAG cases were fewer except in the Moorfields cohort. The Dunfermline (secondary care) and Nottingham (community care) cohorts may be unrepresentative of the wider OHT population because in the UK patients considered to have a moderate or high risk are often kept in secondary care while ocular hypertensives deemed to be very low risk are monitored in primary care (optometry). In addition to shorter follow-up, this may explain the lower incidence of OAG in the Dunfermline (28/188, 14.9%) and Nottingham (5/159, 3.1%) cohorts compared with the DIGS cohort (31/126, 24.6%).

A third limitation was the large proportion of missing values for some predictors, especially CCT. CCT was not routinely collected prior to the publication of the findings of the OHTS study. The OHTS was the first study to document that a thinner central corneal measurement predicts the development of OAG and recommended its measurement in the clinical evaluation of patients with OHT.5

In conclusion, to guide therapeutic intervention and for the prevention of OAG, there is a need for accurate and reliable risk assessment for conversion from OHT to OAG. Despite the limitations of our cohorts, the OHTS-EGPS risk prediction equation generally discriminated well in estimating the 5-year risk of OAG but was not well calibrated in all populations. Therefore, further evaluation in a population-based setting is needed. The equation has not been revised since its original publication, and future research may seek to update the equation to reflect the role and impact of other risk factors in order to improve predictive ability.

Acknowledgments

The authors would like to thank the following for their assistance in the collation of the data for the validation datasets used in this study: Josine van der Schoot and the Rotterdam Ophthalmic Institute (Rotterdam, the Netherlands); Ryo Asaoka (Moorfields Eye Hospital, UK); Karen Armstrong Owen (Queens Medical Centre, Nottingham, UK); and David Holgate and Stuart Dobson (Queen Margaret Hospital, Dunfermline, UK).

References

View Abstract

Footnotes

  • Collaborators Surveillance for Ocular Hypertension Study Group: Jennifer Burr, Adriana Botello, Yemisi Takwoingi, Rodolfo Hernández, Maria Vazquez-Montes, Andrew Elders, Ryo Asaoka, Katie Banister, Josine van der Schoot, Cynthia Fraser, Anthony King, Hans Lemij, Roshini Sanders, Stephen Vernon, Anja Tuulonen, Aachal Kotecha, Paul Glasziou, David Garway-Heath, David Crabb, Luke Vale, Augusto Azuara-Blanco, Rafael Perera, Mandy Ryan, Jonathan Deeks and Jonathan Cook.

  • Contributors JMB, RS and JJD contributed to the conception and design of the study. JMB, AA-B, HGL and JJD obtained funding. JMB, APB, DFG-H, HGL, RS and AJK contributed to data acquisition. YT, APB, JMB, AA-B, HGL and JJD contributed to data analysis. YT conducted statistical analysis under supervision by JJD and drafted the article. YT, APB and AJK provided administrative, technical and material support. All authors contributed to the interpretation of the results, revised the article critically for important intellectual content and gave final approval for publication.

  • Funding This work was part of the Surveillance for Ocular Hypertension study funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme (07/46/02). The funding organisation had no role in the design or conduct of this research.

  • Competing interests DFG-H acknowledges a proportion of his financial support from the Department of Health through the award made by the NIHR to the Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology. DFG-H’s chair at UCL is supported by funding from the International Glaucoma Association.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement All authors agree to allow review of the data by British Journal of Ophthalmology upon request from the corresponding author.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.