Background/aims The prospective, non-interventional ORCA module of the OCEAN study (Observation of Treatment Patterns with Lucentis in Approved Indications) evaluated the qualiy of spectral domain-optical coherence tomography (SD-OCT) image interpretation and treatment decisions by clinicians in Germany and the impact on visual outcomes over 24 months in patients with neovascular age-related macular degeneration (nAMD).
Methods 2286 SD-OCT scans of 205 eyes were independently evaluated by clinicians and reading centres (RCs) regarding signs of choroidal neovascularisation (CNV) activity, including presence of intraretinal fluid, subretinal fluid, and/or increase in pigment epithelial detachments. Agreement between clinicians and RCs was calculated. Treatment decisions by clinicians and the impact on treatment outcomes were evaluated.
Results CNV activity was detected by RCs on 1578 scans (69.0%) and by clinicians on 1392 scans (60.9%), with agreement in 74.9% of cases. Of the 1578 scans with RC detected CNV activity, anti-vascular endothelial growth factor injections were performed by clinicians in only 35.5% (560/1578). In 19.7% of cases (311/1578), lack of treatment was justified by patients request, termination criteria or chronic cystoid spaces without other signs for CNV activity. In 44.8% of cases (707/1578) with RC detected CNV activity, clinicians claimed no treatment was necessary despite having correctly detected CNV activity in about 2/3 of these cases. In 34% of cases with presumed undertreatment, visual acuity declined in the following visit.
Conclusion Although broad agreement on CNV activity parameters was observed between clinicians and RCs, correct identification of CNV activity did not always lead to the initiation of (re-)treatment. To preserve vision over time, correct interpretation of SD-OCT scans and careful retreatment decisions are required.
Trial registration number NCT02194803.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Neovascular age-related macular degeneration (nAMD) is the leading cause of blindness in industrialised countries in the elderly population.1 2 Current guidelines for the management of nAMD recommend the use of intravitreal anti-vascular endothelial growth factor (VEGF) injections.3 4 A number of clinical trials have demonstrated that anti-VEGF treatment for nAMD improves visual outcomes and allows disease control.5–11
Whenever treatment is not performed according to fixed regimes, retreatment decisions are usually based on disease activity criteria detected on optical coherence tomography (OCT). Interpretation of OCT images may be challenging in routine clinical practice. This was evident in the CATT study (Comparison of Age-related Macular Degeneration Treatment Trials), where the evaluations of activity parameters by the central reading centre (RC) were only consistent with 69% of treatment decisions made by the clinicians. Of 30% of cases with discordant results, 95% led to under-dosing.7 Of note, the CATT study was performed in USA and was a clinical Phase 3 study, so it was outside real-world clinical practice.
Non-interventional studies evaluating the treatment of nAMD in real-world settings revealed that the effect of anti-VEGF treatment was less favourable compared with clinical trials.12–16 A number of factors may have contributed to the poorer outcomes, including differences in patient population (eg, possibly worse visual acuity and higher age at baseline), treatment delays, lack of rigorous monitoring and undertreatment due to misinterpretation of imaging data or poor compliance with recommendations.12 15
Differences have been observed regarding visual outcomes of anti-VEGF treatment for nAMD between European countries.13 Despite an initial increase in vision, long term outcomes typically demonstrate slow deterioration in vision over time. The reason for these country-dependent differences remains unclear. Possible explanations include differences in the healthcare systems leading to dissimilar disease management and possible undertreatment (low number of visits/injections), while differences in retreatment criteria and image interpretation may also play a role.13
German ophthalmological societies recommend an initial upload with three intravitreal anti-VEGF injections followed by monthly re-injections according to a pro re nata (as needed) or treat and extend scheme.4 Treatment initiation and decisions on retreatment during follow-up are guided by choroidal neovascularisation (CNV) activity parameters detected on spectral domain OCT volume (SD-OCT) scans through the presence of subretinal fluid (SRF), persistence or increase of diffuse retina thickening, increase of intraretinal cystoid spaces, funduscopy (new retinal haemorrhage) or fluorescein angiography (CNV leakage).4 In contrast, treatment should be terminated if no further improvements are expected due to foveal atrophy and/or fibrosis and if visual acuity declined to <0.05.
The ORCA study compared the evaluation of SD-OCT scans in patients with nAMD undergoing anti-VEGF ranibizumab treatment between clinicians in routine clinical practice in Germany and RCs.17 Treatment decisions of the clinicians were analysed against the background of current treatment guidelines to determine possible undertreatment or overtreatment.4 The overarching goal of the ORCA study was to assess the accuracy of the SD-OCT interpretation and adequacy of treatment decisions, as well as their impact on treatment outcomes during a 24-month review period.
Materials and methods
The data for the present analysis were obtained from the ORCA study as part of the large prospective, multi-centre, non-interventional OCEAN study (Observation of treatment patterns with LuCEntis and real life ophthalmic monitoring, including optional OCT in Approved iNdications).18 The OCEAN study was designed to evaluate the use of OCT in routine clinical practice in Germany. Written informed consent was obtained from all patients
Patients were not included if treated with anti-VEGF injections 3 months prior to study entry or if they had previously received intravitreal steroids. One eye per patient was included. Participants were observed over 24 months in a routine clinical setting at 369 clinical sites.
Detailed information regarding the design of the ORCA module of the OCEAN study has been described previously.16 19 Briefly, physicians at clinical sites (clinicians) performing SD-OCT examinations of adequate quality as part of their routine clinical care, shown by certification, could participate in the ORCA module. To allow adequate evaluation of SD-OCT volume scans, the RCs developed a consensus for minimum quality requirements for colour fundus photography (FP), fluorescein angiography (FA) and SD-OCT imaging. A handbook for image acquisition (including recommended settings for all three modalities) and image data transmission to the RCs was provided to the participating sites. Separate informed consent was required for patients to be included in the ORCA module.
Information collected included best corrected visual acuity (BCVA) at all visits, number and type of anti-VEGF injections and other treatments performed during the study. Imaging data collected included FP, FA and SD-OCT scans at baseline and final visits, and all available SD-OCT scans performed over the 24-month follow-up. Images were evaluated by clinicians as well as three independent RCs blinded to the results of the clinicians (Cologne Image Reading Center, University of Cologne; GRADE Reading Centre, University of Bonn; M3 Reading Centre, St. Franziskus-Hospital, Muenster).
Evaluation of baseline images by the RCs included confirmation of the diagnosis made by the clinicians, analysis of CNV lesion subtype and activity on FA, evaluation of haemorrhage on FP and identification of signs for CNV activity, their location in relation to the fovea and retinal thickness parameters on SD-OCT scans. Both central subfield retinal thickness (CSRT, the average thickness in the central 1 mm area of the fovea that corresponds to the ETDRS subfield 9) and foveal centre point (FCP) thickness (the thickness at the fovea as measured on the very centre B-scan) were used as retinal thickness parameters. Thickness was calculated from the internal limiting membrane to Bruch’s membrane. CSRT values were provided by the automated viewing software, in cases where the scans were considered to be positioned correctly. In case of segmentation errors, manual correction of segmentation lines was performed only if the required time to perform this was considered to be below 2 min—otherwise retinal thickness was evaluated as not assessable. Further, the RCs evaluated whether an imaging parameter was strongly or discreetly visible (figure 1).
In addition, clinicians were asked about their experience with evaluating SD-OCT scans, their level of confidence in their evaluation at each visit, and whether or not they assumed an anti-VEGF injection was indicated for the given patient at these visits.
Agreement between RCs and clinicians regarding baseline diagnosis has been reported previously.16 The current analysis focussed on the subgroup of ORCA participants with RC confirmed nAMD. SD-OCT follow-up evaluations were compared between clinicians in routine clinical practice and RCs. The primary endpoint of the ORCA module was the agreement between RC and clinician for the detection of at least one SD-OCT parameter for CNV activity, defined as presence of intraretinal fluid (IRF), SRF and/or an increase in pigment epithelial detachments (PED). The evaluation process of SD-OCT scans for the ORCA study have been previously described.19 Further, adequacy of treatment indication by the clinician was assessed and possible reasons for suspected undertreatment or overtreatment were evaluated.
Available grading options for presence of all imaging parameters included ‘yes’, ‘no’, ‘questionable’ and ‘cannot grade’. For statistical analysis, ‘questionable’ and ‘cannot grade’ were counted as ‘no’, as no definite presence was attested. The statistical analysis was descriptive, using summary statistics.
A total of 205 patients of the ORCA study were diagnosed with nAMD by clinicians and RCs. For these patients, 2286 follow-up visits with SD-OCT scans and RC as well as clinician evaluation of SD-OCT images were available for analysis. Patients were recruited by 31 clinicians (28 injecting ophthalmologists (90.3%) and three general ophthalmologists (9.7%)). Patients had a mean (±SD) age of 79.9±7.3 and 60.0% were female. A total of 147 patients (71.7%) were categorised as ‘treatment-naïve’ (ie, initial diagnosis occurred within the 3 months before entering the study and no treatment had been initiated), 49 patients (23.9%) as ‘pretreated’ (ie, they had any documented anti-VEGF pre-treatment(s)) and nine patients (4.4%) with missing information on previous treatment as ‘possibly pretreated’. The mean number of SD-OCT scans per patient in the ORCA analysis was 12.2±9.2.
Agreement regarding CNV activity
Of the 2286 available OCT scans from follow-up visits, at least one SD-OCT parameter for CNV activity was detected by the RCs on 1578 scans (69.0%), while the clinicians reported activity on 1392 scans (60.9%). There was agreement in 74.9% of scans (1,712/2,286), with agreement on presence of CNV activity in 1198 scans (52.4%) and agreement on absence of CNV activity in 514 scans (22.5%). In 380 scans (16.6%), the clinician did not detect any sign of CNV activity in contrast to the RC evaluation. In 194 scans (8.5%), the clinician documented signs of CNV activity, whereas the RC did not (table 1).
The presence of IRF and SRF detected by clinicians was confirmed by RC for 73.4% and 77.5% of scans, respectively. A higher proportion (86.9%) of clinician assessments regarding PED increase were confirmed by the RC. However, when clinicians reporting no PED increase, the RC confirmed this assessment only in 18.3% of scans. For IRF and SRF, the RCs confirmed no presence of these parameters in 70.8% and 81.6% of scans, respectively (figure 2).
For IRF, false negative evaluations were documented by clinicians in 314 scans (26.6% of 1181 scans with RC detected IRF), which included 84 scans (26.8%) where the RCs evaluated IRF to be strongly visible. For SRF, false negative evaluations were documented by clinicians in 301 scans (39.5% of 762 scans with RC detected SRF), with RCs having evaluated SRF as strongly visible in 114 of these scans (37.9%). For increase in PED, false negative evaluations were documented by clinicians in 90 scans where PED was reported as stable (58.4% of 154 scans with RC detected increase in PED) and four scans where a decrease in PED was reported (2.6% of 154).
Retinal thickness measurement
CSRT values generated automatically by the software were considered to be correct by the clinicians in 1947 scans (85.2% of 2286), while RCs considered 435 scans (19.0%) to have correct segmentation. Agreement between clinicians and RCs on correct segmentation was documented for 408 scans. Segmentation errors were manually corrected by RCs in 1566 scans (68.5%) and in 115 scans (5.0%) by clinicians.
CSRT and FCP parameters for all assessed scans are presented in figure 3. For the 1791 scans where both RCs and clinicians provided CSRT values, the mean difference per patient of the calculated CSRT was 41.4±96.0 µm higher when measured by RCs than by clinicians. For the 2140 scans where both RCs and clinicians provided FCP thickness values, the mean difference per patient in the calculated FCP thickness was 24.7±120.5 µm higher by RCs than by clinicians.
Evaluation of possible overtreatment or undertreatment
In 1578 scans, RCs detected signs of CNV activity. Anti-VEGF injections were performed by the clinicians based on 560 of these scans (35.5%). In 311 scans (19.7%), the decision of the clinician to not perform treatment was justified in the opinion of the RC, for example, due to the presence of chronic cystoid spaces only, because of documented patient’s wish or because no further benefit was expected from treatment (termination criteria fulfilled). However, for the remaining 707 scans resulting in no injection despite RC detected CNV activity (44.8%), no obvious justification was identified by the RC. In only 31.3% of these scans, false negative evaluations for CNV activity were performed by the clinicians and thus misinterpretation of images may have led to the suspected undertreatment. In the remaining 68.7% of scans, the clinician correctly identified IRF, SRF or PED increase as sign for CNV activity but deemed treatment not necessary in discordance with the RC (figure 4).
For 243 of the 707 scans with potential undertreatment (34.4%), deterioration in BCVA was observed at the subsequent visit. This visual acuity loss was documented in 114 patients managed by 22 clinicians.
When examining the impact of the individual clinicians, a total of 142 patients were found to be at risk of undertreatment from 23 clinicians, while 74 patients were found to be at risk of overtreatment from 18 clinicians.
Physicians’ experience and confidence in the SD-OCT evaluations
Most clinicians (23/29, 79.3%) stated that they had more than 2 years of experience in OCT diagnostics, three clinicians (10.3%) had 1–6 months of experience and three clinicians (10.3%) had 1–2 years of experience. When asked about their level of confidence in the evaluation of each SD-OCT scan, most clinicians stated that they were confident (1319/2286 scans; 57.7%) or very confident (692/2286 scans; 30.3%) of their evaluations.
Although several clinical trials have demonstrated that nAMD treatment with intravitreal anti-VEGF injections improves visual acuity and prevents vision loss over time, in non-interventional studies the effects of anti-VEGF treatment in real-life analyses have been less visible. To date, the reason for this remains unclear. Inaccurate diagnosis and interpretation of imaging findings may be an important reason for suboptimal visual outcomes in OCT-guided treatment regimen in real life, due to subsequent undertreatment. The impact of misdiagnosis at baseline has been reported previously for the ORCA study and has been proven to impact visual acuity outcomes.16 Challenges in disease activity assessments are likely even more important and may have a great impact on vision outcome over time.
The ORCA study aimed to analyse the agreement in interpretation of SD-OCT scans and consequent treatment decisions between clinicians in routine clinical practice in Germany and RCs, as well as their impact on treatment outcomes during a 24-month follow-up.16 In the present study, clinicians and RCs relatively frequently agreed in the detection of at least one CNV activity parameter on SD-OCT scans (74.9% of scans). These results are similar in comparison to the CATT study, revealing differences in OCT interpretation between clinicians and RCs in 69% of cases.7 Agreement for single parameters of CNV activity in ORCA was better for SRF followed by IRF and PED. Nonetheless, in about one-quarter of scans, the RCs’ evaluation disagreed with the clinicians. In only 8.5% of scans, clinicians documented signs of disease activity that were not confirmed by the RCs. Treatment decisions triggered by this evaluation may be unnecessary and increase the risk of side effects and put an avoidable strain on the healthcare system. In 16.7% of scans, clinicians overlooked signs of disease activity reported by the RCs. False negative evaluation of CNV activity may lead to undertreatment and thus avoidable vision decrease. However, in the majority of scans with false negative evaluations for CNV activity, IRF and/or SRF were considered by the RCs to be either outside of the foveal centre or only discreetly visible. Thus, while a negative effect on vision may not be ruled out, the impact in these cases may be low.
Interestingly, suspected undertreatment in the present study was not mainly triggered by false negative evaluation of parameters for CNV activity on SD-OCT scans, but by the decision of the clinicians that treatment was not necessary despite the correct detection of IRF, SRF and/or increase in PED. Specifically, for a considerable number of scans (64.5%) with CNV activity according to the RCs, no treatment was administered. Among scans with CNV activity, only 311 (19.7%) provided justifications for the ‘watch and wait’ approach according to the RC, for example due to presence of chronic cystoid spaces only or no further expected benefit from treatment. However, in most scenarios with potential undertreatment, no obvious justification could be identified by the RCs. One may speculate that presence of rather small amounts of fluid or only extrafoveal fluid may have been considered irrelevant, however this has not been documented. In our study, 34.4% of scans without treatment despite RC detected CNV activity led to lost vision in patients at the subsequent visit, thus a consequent treatment regime seems important to preserve vision over time. Unlike in other countries, access to ranibizumab or appropriately scheduled appointments for treatment in Germany is not problematic and thus seems a rather unlikely explanation. It has been shown by the AURA13 14 (Study to Assess the Effectiveness of Existing Anti vascular Endothelial Growth Factor in Patients witn wet Age-related Macular Degeneration) and WAVE12 (Lucentis in Wet AMD: Evaluation of Visual Acuity and Quality of Life) studies, as well as in another module of the OCEAN study,20 that delayed or reduced ranibizumab injections lead to a decreased improvement of visual acuity. The importance of correct and timely treatment is emphasised by the fact that vision loss was documented at the subsequent visit for more than one-third of the scans with potential undertreatment in the present study.
One approach that has been suggested to improve identification of CNV activity on SD-OCT scans is the implementation of artificial intelligence.21 Classic machine learning and deep learning algorithms have produced highly sensitive and specific interpretations of clinical findings and could support a standardised and less error-prone diagnosis procedure.22 23 Challenges however include the large number of different OCT systems in place, proprietary data formats and regulatory requirements.24 Therefore, adequate training of clinicians in SD-OCT interpretation and treatment indication remains important.
The ORCA study has some limitations due to the non-interventional design. Documentation might not be as stringent as in clinical trials. A minimum standard of imaging was required for participation, and clinicians had to disclose image interpretation and treatment decisions to the RCs, therefore rather experienced and confident clinicians may be over-represented in this real-world setting. In addition, individual reasons for suspended treatment despite correctly detected signs for CNV activity have not been documented.
The ORCA study demonstrated possible undertreatment in many cases although in most cases, clinicians accurately identified signs for CNV activity on SD-OCT scans. The study provides strong evidence that misinterpretation of SD-OCT scans not only results in undertreatment, but also in visual loss over time. Lack of treatment appears to also occur even when correct identification of IRF, SRF and/or PED was documented. The reason for this latter observation remains unclear and should be identified in future studies. Our study showed that RC evaluation may be valuable, for example in the context of quality assurance. Not only is adequate training or possibly future artificial intelligence assistance in SD-OCT interpretation needed, but also precise guidelines for retreatment and adherence to these is recommended to preserve vision over time.
We thank Klaus Laschke (statistics), Susanne Faber (project management), Bianca Arendt (medical writing), Anthi Trasta (medical writing), Daniela Mueller (medical writing) and Catherine Mason (medical writing), all Kantar GmbH, Munich, Germany, for their assistance, which was funded by Novartis Pharma GmbH, Nuremberg, Germany. The participants of the ORCA Group are listed in the online supplementary material.
Contributors SL (study design, data collection, data analysis, revision of the manuscript), GS (study design, data collection, data analysis, revision of the manuscript), CKB (data collection, data analysis, revision of the manuscript), TS (data collection, data analysis, revision of the manuscript), FZ (data collection, data analysis, revision of the manuscript), JV (study design, data analysis, revision of the manuscript), MK (data analysis, revision of the manuscript), BK (study design, data analysis, revision of the manuscript), FGH (study design, data analysis, revision of the manuscript), DP (study design, data analysis, revision of the manuscript), SS-V (study design, data collection, data analysis, revision of manuscript). All authors read and approved the final manuscript.
Funding This study was developed and funded by Novartis Pharma GmbH, Nuremberg, Germany.
Competing interests SL received personal fees from Novartis, Heidelberg Engineering, Carl Zeiss Meditec, Allergan, Bayer, and was member of an advisory board for Novartis. GS received personal fees from Bayer Healthcare, Novartis, Zeiss, OD-OS and Allergan Pharma. CKB has received grants from Heidelberg Engineering, Newtricious, Novartis, Zeiss Meditec; and personal fees from Heidelberg Engineering, Newtricious and Novartis. TS received personal fees from Novartis. FZ received grants and personal fees from Alimera, Allergan, Bayer Healthcare, Biogen and Roche, grants from Clearside, personal fees from Boehringer Ingelheim, Novartis, NovoNordisk and MSD Sharp & Dohme. JV and MK are employees of Novartis. BK received grants from Novartis and personal fees from Novartis, Pfizer and Bayer. FGH received grants from Acucela, Alcon, Allergan, Bayer, Genentech, Heidelberg Engineering, Novartis, Ophthotech, Roche and personal fees from Allergan, Bayer, Genentech, Novartis, Ophthotech; he was a member of advisory boards for Acucela, Allergan, Avalanche, Bayer, Genentech, Heidelberg Engineering, Johnson & Johnson, Novartis, Ophthotech and Roche. DP received grants from Novartis, Ophthotech, Acucela, Genentech, Roche, Bayer and personal fees from Novartis, Bayer, Allergan; he was a member of advisory boards for Novartis, Bayer and Allergan. SS-V received grants and non-financial research funding from Acucela, Alcon/Novartis, Allergan, Bayer, Bioeq/Formycon, Centervue, Galimedix, Genentech/Roche, Heidelberg Engineering, Optos, Carl Zeiss MediTec, and personal fees and honoraria from Alcon/Novartis, Bayer, Genentech/Roche and Carl Zeiss MediTec.
Patient consent for publication Not required.
Ethics approval The OCEAN study was approved by the ethics committee of the Eberhard-Karls-University, Tuebingen, Germany. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement The datasets generated during and/or analysed during the current study are not publicly available due to German data protection law but are available from the corresponding author on reasonable request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.