Article Text
Abstract
Aim: To analyse variations in rates of severe retinopathy of prematurity (ROP) among neonatal intensive care units (NICUs) in the Australian and New Zealand Neonatal Network (ANZNN), adjusting for sampling variability and for case mix.
Methods: 25 NICUs were included in the study of 2105 infants born at less than 29 weeks in 1998 and 1999, who survived to 36 weeks postmenstrual age and were examined for ROP. The observed NICU rates of severe ROP were adjusted for case mix using logistic regression on gestation, weight for gestational age and sex, and for sampling variability using shrinkage estimates. The corrected rate in the best 20% of NICUs was identified and NICU variations in rates were compared with those in 2000–1.
Results: The overall (unadjusted) rate of severe ROP in the NICUs was 9.6% (interquartile range 5.4−12.8%). After adjusting for both case mix and sampling variability there remained significant variation among the NICUs. 20% of NICUs had a rate of severe ROP ⩽5.9%. Variation in rates among NICUs showed a similar pattern in both time periods. If the overall network rate was reduced to 5.9%, the 20th centile of the adjusted rates, there would be 79 fewer cases in a 2 year period, in contrast with 26 fewer if rates in the two units with excess rates improved to the average.
Conclusions: Considerable variation in rates of severe ROP among NICUs remained after adjustment for case mix and sampling variability. These data will facilitate investigation of potentially better practices associated with a reduced risk of severe ROP.
Statistics from Altmetric.com
 ANZNN, Australian and New Zealand Neonatal Network
 CRIB, Clinical Risk Index for Babies
 GA, gestational age
 IQR, interquartile range
 NICU, neonatal intensive care unit
 PMA, postmenstrual age
 ROP, retinopathy of prematurity
 SNAPPE, Score for Neonatal Acute Physiology
 retinopathy of prematurity
 premature infants
 quality improvement
 ANZNN, Australian and New Zealand Neonatal Network
 CRIB, Clinical Risk Index for Babies
 GA, gestational age
 IQR, interquartile range
 NICU, neonatal intensive care unit
 PMA, postmenstrual age
 ROP, retinopathy of prematurity
 SNAPPE, Score for Neonatal Acute Physiology
 retinopathy of prematurity
 premature infants
 quality improvement
Since the first description of retrolental fibroplasia by Terry in 1942, a notable feature of the condition (now known as retinopathy of prematurity, ROP) has been the variation in incidence among hospitals.^{1} Recent hospital cohorts^{2,}^{3} and population based reports^{4–}^{8} continue to demonstrate widespread variability in reported incidence and severity of ROP.
Such variation in outcome, across different specialties and disease entities, has been the subject of considerable debate. Differences in outcome ascertainment, case mix, and sampling variability, as well as different clinical practices may all contribute to the observed differences.^{9} In order to be able to identify NICUs with significantly better outcomes, and thus investigate potentially better practices associated with those outcomes, it is necessary to control for these sources of variation. No previous study has taken this approach to reduction in the incidence of severe ROP.
The Australia and New Zealand Neonatal Network (ANZNN) consists of all 29 level III neonatal intensive care units (NICUs) in Australia and New Zealand. Since 1995 a dataset of 60 perinatal variables, using agreed definitions, has been collected prospectively on all infants of less than 32 weeks gestation or birth weight less than 1500 g. Both countries have a regionalised system of neonatal intensive care, meaning that the ANZNN dataset for infants with gestation less than 29 weeks represents essentially population based data. Previously, we analysed data on 21 antenatal and intrapartum variables for all infants <29 weeks gestation born in the 2 year period 1998–9 and who survived to 36 weeks postmenstrual age (PMA) to identify risk factors for severe (stage 3 or more^{10}) ROP.^{11} That study showed that decreasing gestation, lower birth weight for gestation, and male sex were significantly associated with increased risk.^{11}
We have now analysed the differences in severe ROP among NICUs of the ANZNN, adjusting for differences in case mix and sampling variability, using our previously reported methodology.^{9} Our aim was to identify true variations in severe ROP in order to provide a framework for reducing the overall network incidence of severe ROP.
METHODS
The four children’s hospitals out of the 29 NICUs in the ANZNN were excluded from the study since they represent a different population of infants (all outborn and many requiring surgery). The remaining 25 NICUs included 2830 infants born at <29 weeks in 1998–9, of whom 2286 (80.8%) survived to 36 weeks PMA. The 181 (7.9%) infants who did not have an eye examination reported were excluded, leaving 2105 eligible infants.
To analyse variation in outcome rates among NICUs we compared the observed frequency (O) of severe ROP in each unit with the expected frequency (E).^{9} The unadjusted value of E for each NICU is simply the overall rate of severe ROP for all NICUs multiplied by the number of eligible admissions to that NICU. To adjust for bias caused by case mix, we used the logistic regression model to predict the probability of severe ROP for each infant from its gestation, weight for gestational age and sex.^{11} The adjusted value of E for each NICU is the sum of the predicted probabilities for all infants in that NICU.
We also allowed for the effect of sampling variability, which is smaller when the outcome rate is closer to 0% or to 100%, assuming the rates have a binomial distribution. It is also smaller for larger NICUs, because the standard error of the estimated rate decreases as sample size increases. Because of sampling variability the observed rate varies about the true rate for each NICU. We allowed for this by calculating shrinkage estimators using a Bayesian approach. Details are given elsewhere^{9} but, briefly, each NICU is assumed to have a true underlying rate of severe ROP. Shrinkage estimators use data from all NICUs to improve the estimate of the true rate of severe ROP for each individual NICU, by shrinking the observed rate towards the overall rate for all units. The shrinkage is greater for smaller NICUs, so shrinkage estimates are more conservative and less variable than the observed rates. The shrinkage estimates were obtained using maximum likelihood methods assuming a gamma Poisson model, which also gave an estimate, v, of the standard deviation of the true incidence ratios (O/E) for all NICUs.^{12}
The corrected excess (O − E) number of infants with severe ROP was calculated for each NICU using the shrunken estimators. The 95% control limits were obtained similarly, to show the limits within which this excess should lie when the expected number of cases is E.^{9}
To inform quality improvement, we explored the benefit of a networkwide approach rather than targeting NICUs performing poorly. The 20th centile was estimated as the (n+1)/5th value of the adjusted rates of severe ROP for all NICUs sorted in ascending order. This is used as an achievable estimate of “best practice,” because it is already achieved by 20% of NICUs. The “centile gain” is the number of potential cases of severe ROP that could be prevented in a 2 year period if the overall NICU rate was reduced to the 20th centile rate. This was compared with the “outlier gain,” the reduction in the number of cases that would result if NICUs that were exceeding their 95% limits improved to their expected number of cases.
The same methods were applied separately to data for a similar cohort born in 2000–1 (that is, infants of <29 weeks gestation, registered with ANZNN in 2000–1, who survived to 36 weeks PMA and were examined for ROP) for comparison (n = 2277) (full data for 2000–1 not shown).
We also explored the potential measurement bias resulting from the failure to examine eligible infants. Kendall’s rank correlation coefficient was used to assess the relations between the rate of severe ROP and the observed mortality rate (that is, the proportion of infants of <29 weeks gestation who died before 36 weeks PMA), the number of infants of <29 weeks gestation alive at 36 weeks PMA, and the percentage of these eligible infants not examined for ROP per NICU. In addition, sensitivity analysis was performed using best and worst case scenarios (worst case treating babies not examined for ROP as cases of severe ROP, best case as no ROP^{11}).
Lastly, the NICUs were surveyed on their clinical practice with respect to indirect ophthalmoscopy. All NICUs in the ANZNN confirmed that examination guidelines^{13,}^{14} with respect to timing and technique, including adequate pupil dilatation, were followed and that infants are screened until the retina is fully or nearly fully (extreme zone 3) vascularised. Examination findings are reported using the International Classification of ROP.^{10}
Approval for the project was given by Royal Prince Alfred Hospital ethics review committee and University of Sydney human ethics committee.
RESULTS
In 1998–9 ROP of any stage was reported in 42% of examined infants, stage 3 or more in 9.6%, and 57% of the latter received treatment by laser or cryotherapy. The unadjusted rates of severe ROP for individual NICUs ranged from 0% to 25% (column 4, table 1), interquartile range (IQR) 5.4–12.8%. The effects of both the shrinkage and case mix adjustment are shown in table 1. Adjusting for sampling variability alone (column 6) has a relatively minor impact on the order of the NICUs (except for “H”). Adjusting for case mix has a more marked effect on the order, most notably for “P” and “T.” The relative impact of the adjustments is best demonstrated graphically. Figure 1 compares the observed rate of severe ROP with the rates adjusted for (a) case mix only and (b) both case mix and sampling variability (corrected rates).
Figure 2 shows the number of infants involved in this variation by depicting the corrected excess (O − E) number of severe ROP cases for each NICU in 1998–9 (black diamonds), with the NICUs ordered by decreasing number of examined infants and the upper and lower 95% control limits shown. NICU “X” had almost 20 more cases of severe ROP than expected over the 2 year period. NICU “Y” had six excess cases and NICU “B” had five fewer cases of severe ROP than expected.
For infants in the 2000–1 cohort, the overall rate of severe ROP was 10.2% (60% of these infants receiving treatment) and the variability among NICUs increased slightly (fig 2, open squares). There were an additional two NICUs (“O” and “G”) with fewer than expected cases and another two NICUs (“R” and “N”) with more than expected. Individual unit performance for this outcome can be easily compared over the two time periods using figure 2 because the NICUs are in the same order. The NICU with almost 20 more cases than expected in 1998–9 had 25 excess cases in 2000–1.
Estimates of the variation (v) among the NICUs’ true incidence ratios are given in table 2 for three different scenarios: with adjustment for none, two, and three predictor variables. The variability among NICUs actually increased as we adjusted for more variables. Adjusting for the three predictors (gestational age, birth weight for gestational age, and sex), the best 20% of NICUs were estimated to have a rate of 5.9% or less while the overall rate in this period was 9.6%. If the network rate was reduced to 5.9%, there would be 79 fewer cases of severe ROP over a 2 year period (“centile gain”), in contrast with only 26 fewer cases by taking the “outlier gain” approach—that is, by improving the outcome in the two NICUs that were in excess of their 95% limits.
There was no relation between the observed rate of severe ROP and a NICU’s mortality rate before 36 weeks PMA (Kendall’s rank correlation, t = 0.10, p = 0.47), or size of NICU, as judged by the number of eligible infants (T = 0.04, p = 0.78). Measurement bias could also result from excluding those infants not examined for ROP, with the potential for NICUs with high rates of “not examined” to have low rates of severe ROP. Figure 3 displays these data with the size of each point proportional to the number of infants in the cohort from each NICU. There was a significant negative correlation between the rate of severe ROP and the rate of “not examined” (Kendall’s rank correlation, T = −0.30, p = 0.04). The sensitivity analysis using the worst case scenario—that is, assuming all nonexamined infants had severe ROP, resulted in an additional two NICUs having corrected excess above their 95% limits. However, these NICUs had a greater than average proportion of larger, more mature infants, who have a relatively low risk of severe ROP.
DISCUSSION
This study has demonstrated that even when data are corrected for case mix and sampling variability there remains considerable variation in rates of severe ROP among 25 NICUs in the ANZNN.
Our analysis did demonstrate a significant relation between the percentage of infants not examined for ROP and the rate of severe ROP. NICUs with lower rates of ROP tended to have a greater proportion of eligible infants not examined. Overall, only a small proportion of eligible infants (7.9%) were not examined and our previous analysis showed the majority of these were larger and more mature, with 82% being in the upper quartile for birth weight and 75% being born at 27 weeks gestation or beyond.^{11} One possibility is that NICUs with lower rates of ROP in infants most at risk were more likely not to examine larger and more mature infants. However, these data do emphasise the need for all infants to have appropriate screening for ROP according to the national guidelines.
To examine variation in practice among NICUs, we focused on antenatal and intrapartum variables that affect the infant’s condition before arrival in the NICU when adjusting for case mix. One possible source of variation is that some infants were intrinsically sicker than others. A number of scores have been developed as a measure of severity of illness, including the Clinical Risk Index for Babies (CRIB)^{15} and Score for Neonatal Acute Physiology (SNAPPE).^{16,}^{17} The ANZNN does collect data to calculate the CRIB score but not the SNAPPE. However, the CRIB score includes birth weight and gestation, which were both used in our logistic regression model, together with highest and lowest inspired oxygen in the first 12 hours of life, which may be influenced by care practices. Additionally, both scores are reported to predict hospital mortality better than they predict morbidity.^{18}
Parry et al^{19} reported on a longitudinal study of mortality among 2671 infants with birth weight <1500 g admitted to one of nine NICUs in the United Kingdom between 1988 and 1994. While in most years there were no significant differences in outcome among hospitals, the apparent performance of individual hospitals fluctuated substantially from year to year. One interesting feature of the present study is that when the same model was applied to 2000–1 data most units had similar results, in particular the two NICUs that had an excess of cases in 1998–9 still had this excess in the later period. This does suggest the possibility that there may be some systematic or consistent element(s) in clinical practice in these NICUs, which may be relevant. Equally, the one NICU with fewer than expected cases in 1998–9, continued in this situation in 2000–1. A focus on systemic quality improvement measures, aimed at reducing the incidence of severe ROP in all NICUs, is likely to be more productive than an approach that focuses solely on poorly performing NICUs. In particular, reducing the overall network rate to the 20th centile rate is likely to prevent more cases of severe ROP in 2 years (79) than if the two NICUs with excess rates improved to their expected number of cases (26 cases prevented).
One source of variation we are unable to correct for is observer differences in the assessment of retinal changes during indirect ophthalmoscopy. Such measurement bias is harder to quantify for ROP than for some other neonatal outcomes, such as intraventricular haemorrhage. At present it is not straightforward or practical for independent observers to audit the results of individual indirect ophthalmoscopy examinations, although the development and availability of contact digital retinal cameras may change this situation in the future.^{20} However, the NICUs in the ANZNN are all served by experienced ophthalmologists, who follow explicit examination standards guidelines,^{13,}^{14} which should help to minimise this source of variation.
In conclusion, we have analysed data on the incidence of severe ROP among NICUs in the ANZNN and adjusted for both sampling variation and case mix to explore variations in rates among hospitals. Greater reductions in rates of severe ROP could be achieved if the overall rate was reduced to the 20th centile than if the focus were on outliers with excess rates. There has been much recent debate about optimal oxygen therapy in extremely premature infants^{21,}^{22} and two recent trials^{23,}^{24} have compared oxygen saturation targets after the first few weeks of life in these infants. Other studies have found a relation between greater variability of oxygen levels and an increased risk of severe ROP,^{25,}^{26} while strategies to decrease some of this variability have also been reported.^{27} These and other evidence based factors should be explored in efforts to achieve networkwide improvement in rates of severe ROP.^{28}
APPENDIX
The ANZNN Advisory Committee and Executive (*) members:
Australia: Centre for Perinatal Health Services Research, NSW: David HendersonSmart*; Flinders Medical Centre, SA: Peter Marshall; John Hunter Hospital, NSW: Chris Wake; King Edward Memorial and Princess Margaret Hospitals, WA: Noel French, Ron Hagan and Karen Simmer; Launceston General Hospital, Tas: Chris Bailey; Liverpool Health Service, NSW: Robert Guaran; Mater Mother’s Hospital, Qld: David Tudehope; Mercy Hospital for Women, Vic: Andrew Watkins; Monash Medical Centre, Vic: Kaye Bawden*, Andrew Ramsden, Victor Yu; National Perinatal Statistics Unit, NSW: Paul Lancaster*; Nepean Hospital, NSW: Lyn Downe; Newborn Emergency Transport Service (Vic): Michael Stewart; NSW Newborn and Paediatric Emergency Transport Service: Andrew Berry; Perinatal Research Centre, Qld: Paul Colditz; Royal Children’s Hospital, Vic: Linda Johnstone, Peter McDougall; Royal Darwin Hospital, NT: Charles Kilburn; Royal Hobart Hospital, Tas: Peter Dargaville; Royal Hospital for Women, NSW: Kei Lui; Royal North Shore Hospital, NSW: Jennifer Bowen; Royal Prince Alfred Hospital, NSW: Nick Evans Royal Women’s Hospital, Qld: David Cartwright*; Royal Women’s Hospital, Vic: Lex Doyle, Colin Morley, Neil Roy; Sydney Children’s Hospital, NSW: Barry Duffy; The Canberra Hospital, ACT: Graham Reynolds; The Children’s Hospital at Westmead, NSW: Robert Halliday; The Townsville Hospital, Qld: John Whitehall; Western Australia Neonatal Transport Service: Jenni Sokol; Westmead Hospital, NSW: William TarnowMordi; Women’s and Children’s Hospital, SA: Ross Haslam; Deborah Donoghue is the ANZNN Coordinator;
New Zealand: Christchurch Women’s Hospital: Nicola Austin; Christchurch School of Medicine: Brian Darlow*; Dunedin Hospital: Roland Broadbent; Gisborne Hospital: Graeme Lear; Hastings Hospital: Jenny Corban; Hutt Hospital: Robyn Shaw; Middlemore Hospital: Lindsay Mildenhall; National Women’s Hospital: Carl Kushell; Nelson Hospital: Peter McIlroy; Palmerston North Hospital: Jeff Brown; Rotorua Hospital: Stephen Bradley; Southland Hospital: Paul Tomlinson; Taranaki Hospital: John Doran*; Tauranga Hospital: Hugh Lees; Timaru Hospital: Philip Morrison; University of Auckland: Jane Harding; Waikato Hospital: David Bourchier; Wairau Hospital: Ken Dawson; Wanganui Hospital: John Goldsmith; Wellington Women’s Hospital: Vaughan Richardson; Whakatane Hospital: Chris Moyes; Whangarei Hospital: Peter Jankowitz.
Acknowledgments
Supported by the National Health and Medical Research Council of Australia (grant 211088 to DRHS, DAD, JMS, and NJE) and the Centre for Perinatal Health Services Research, University of Sydney, Sydney, Australia.
Footnotes

↵* See appendix
Request permissions
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.