Objectives To investigate inter-reader agreement on five severity levels of central vascular changes (none, mild, moderate, severe pre-plus disease, plus disease) and aggressive posterior retinopathy of prematurity (ROP), and to see whether an unintended shift in indication for treatment occurred.

Methods Four international ROP readers participated. Before the grading of the photographs, the readers were informed that a high proportion of advanced ROP cases were included. In total, 243 photographs/948 quadrants were available from 136 infants. As a standard series of photographs was available, grading was performed under optimised conditions.

Results The four readers agreed on the quadrant scores of only 70 (7.38%) of the 948 quadrants—that is, on 1, 5, 15, 4 and 45 quadrants for scores 0, 1, 2, 3 and 4, respectively. The mean scores differed systematically between the readers (permutation test, p<0.0001). Agreement on presence of aggressive posterior ROP from all four readers was not obtained for any of the photographs. Readers scored plus disease in at least two quadrants in 95.5% of the eyes for which treatment was indicated. All four readers agreed on the scoring of indication for treatment for 195 eyes (80.2%); however, treatment was only recommended in 18 (7.4%) eyes. One reader was found to differ systematically from the others in indicating treatment (Rasch analysis; p=0.0001). Finally, a significant shift in indication for treatment occurred between birth period 2000–2002 and 2003–2006 (Mann–Whitney rank sum test, p<0.001).

Conclusions Inter-reader agreement on central vascular changes is poor, especially when based on more than two rating categories. The subjective nature of diagnosing such vascular changes possibly resulted in earlier treatment of preterm infants in Denmark over the entire study period (1997–2006). The recent increased incidence of treated infants in Denmark is, at least in part, explained by a significant shift in indication for treatment.