Article Text
PDF
Abstract
Background/rationale Artificial intelligence (AI)-based clinical decision support tools, being developed across multiple fields in medicine, need to be evaluated for their impact on the treatment and outcomes of patients as well as optimisation of the clinical workflow. The RAZORBILL study will investigate the impact of advanced AI segmentation algorithms on the disease activity assessment in patients with neovascular age-related macular degeneration (nAMD) by enriching three-dimensional (3D) retinal optical coherence tomography (OCT) scans with automated fluid and layer quantification measurements.
Methods RAZORBILL is an observational, multicentre, multinational, open-label study, comprising two phases: (a) clinical data collection (phase I): an observational study design, which enforces neither strict visit schedule nor mandated treatment regimen was chosen as an appropriate design to collect data in a real-world clinical setting to enable evaluation in phase II and (b) OCT enrichment analysis (phase II): de-identified 3D OCT scans will be evaluated for disease activity. Within this evaluation, investigators will review the scans once enriched with segmentation results (i.e., highlighted and quantified pathological fluid volumes) and once in its original (i.e., non-enriched) state. This review will be performed using an integrated crossover design, where investigators are used as their own controls allowing the analysis to account for differences in expertise and individual disease activity definitions.
Conclusions In order to apply novel AI tools to routine clinical care, their benefit as well as operational feasibility need to be carefully investigated. RAZORBILL will inform on the value of AI-based clinical decision support tools. It will clarify if these can be implemented in clinical treatment of patients with nAMD and whether it allows for optimisation of individualised treatment in routine clinical care.
- imaging
- macula
- neovascularisation
- retina
- treatment medical
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Background and rationale
Neovascular age-related macular degeneration (nAMD), a late form of AMD, is characterised by neovascularisation, retinal vascular leakage and fluid accumulation. The standard of care for nAMD is repeated intravitreal injection with anti-vascular endothelial growth factor (anti-VEGF) agents including the licensed aflibercept, ranibizumab and, the recently approved, brolucizumab.1 2 However, also bevacizumab is used in an off-label fashion as intravitreal injected anti-VEGF.
Evidence from previous robust randomised controlled trials using aflibercept and ranibizumab with treat-and-extend protocols has indicated an average of approximately eight injections/year as the treatment frequency to achieve optimal outcomes.3 However, non-interventional observational studies have shown that the average number of injections and treatment visits observed in real-world clinical situations are significantly lower than expected.4 For example, in the observational OCEAN study, the 3726 patients with nAMD received only an average of 4.47 ranibizumab injections during an observational time of 12 months.5 This was notably lower than what would have been expected based on available randomised controlled trials.6 One contributing factor to this undertreatment may be linked with delayed retreatment decisions made by physicians.7 Time constraints, arising from an overwhelmed healthcare system, may result in insufficient assessment of disease activity via critical analysis of optical coherence tomography (OCT) scans of the retina, which invariably informs and influences retreatment decisions.7–11 This was evident in the ORCA substudy of the OCEAN study, where in 380 (16.6%) out of 2286 OCT readings from 205 eyes, physicians did not detect any sign of choroidal neovascularization (CNV) activity in contrast to the evaluation by a central reading centre.5 Furthermore, the CATT study reported in 2782 (29.4%) out of 9455 examinations inconsistencies of missed treatments, where the reading centre detected pathological fluid, but the patient was not treated.12
By using artificial intelligence (AI) to assist the critical analysis of OCT scans, a faster and more sensitive identification of disease activity might be achieved, which could translate directly into treatment decisions. Additional information currently not available in routine practice gleaned from the AI-assisted OCT analysis such as volumetric measures of fluid within the different retina compartments may in fact allow for better and clearer differentiation and decision for treatment. Here, especially because of the fact that volumetric measurements are currently not performed and manual review of multiple OCT segment cuts is needed to understand the three-dimensional (3D) aspect.13 This may contribute to address one of the multiple root causes, leading to undertreatment issues faced in the real-world clinical setting. AI applied at the point of care would allow for a personalised treatment approach by providing real-time readout support during the patient encounter.
Based on these understandings, we hypothesise that Discovery (RetinAI Medical AG, Bern, Switzerland), 14–16 an ophthalmology image management platform embedded with advanced AI technology for analysing and enriching 3D OCT scans, will add specific supportive information for a faster and more precise evaluation of anatomical signs of disease activity particularly in less specialised centres. The system will detect, identify, highlight and quantify pathological fluid compartments associated with neovascular lesion activity to the treating physician in real time. To generate robust evidence to support this hypothesis, we will investigate in this multicentre observational study, the extent to which enrichment of 3D OCT scans with segmentation information (and volumetric information) can optimise and support disease activity assessment in patients treated for nAMD. Furthermore, the study aims to investigate how well Discovery is useful to physicians and implementable in selected hospitals and whether it can optimise the clinical workflow.
Methods
Study design
RAZORBILL (Title: A non-interventional study to assess the influence of automated optical coherence tomography (OCT) image enrichment with segmentation information on disease activity assessment in patients treated with licensed anti-VEGF injections; clinicaltrials.gov identifier NCT04662944) is an observational, multicentrer, multinational, open-label study designed with the objective to primarily assess the influence of automated 3D OCT scan enrichment with segmentation information on disease activity assessment in patients with nAMD treated with either brolucizumab, ranibizumab or aflibercept according to the respective label. RAZORBILL was designed in accordance to the SPIRIT-AI guidelines.17
The study comprises of two phases (see figure 1):
Clinical Data Collection (Phase I)
OCT Enrichment Analysis (Phase II).
Flow diagram of RAZORBILL study. Collection of clinical data (including 3D OCT scans) will be performed during routine clinical care (phase I). The data will be stored in the Discovery platform. Analysis of 3D OCT scans will be conducted after 3D OCT scans are partially enriched via segmentation algorithms (phase II). 3D, three-dimensional; OCT, optical coherence tomography.
In phase I, clinical and imaging data will be collected in a real-world clinical setting. An observational study design, which enforces neither strict visit schedule nor mandated treatment regimen, was chosen as an appropriate design. Hence, this non-interventional study does not mandate a therapy protocol, diagnostic/therapeutic procedure or a visit schedule. The diagnostic or monitoring procedures are only those ordinarily applied to the therapeutic strategy and to routine clinical care and will take place as per the site investigator’s discretion. This implies that no standardised OCT imaging protocol will be used to ensure that collected 3D OCTs, used for phase II, are representative of OCTs recorded in routine clinical care.
In phase II, 3D OCT scans, collected in phase I, will be reviewed by a panel of ophthalmologists for disease activity to address the main study objective. Each 3D OCT scan will hereby be reviewed, once enriched by previously designed segmentation algorithms and once in its original (i.e., non-enriched) form in a crossover design.
Phase II will start with a time delay with respect to phase I, so that no information will be fed back into phase I, and no real-world treatment decision will be influenced by any readouts or analysis performed in phase II.
Clinical data collection (phase I)
Study population
The study will enrol 720 patients with nAMD who are either treatment-naive or currently treated with either aflibercept, brolucizumab or ranibizumab according to the respective drug label. Patients will be enrolled in approximately 20 centres, across five countries—Canada, Germany, Ireland, Italy and Spain. The prospective observation and data collection period per patient will be 12 months. Most important inclusion and exclusion criteria are summarised in box 1.
Most important inclusion and exclusion criteria
Inclusion criteria
Diagnosis of neovascular age-related macular degeneration (nAMD).
Male and female patients with ≥40 years of age at baseline with signed written informed consent.
Patients for whom a therapy with aflibercept, brolucizumab or ranibizumab is medically indicated.
Intraretinal and/or subretinal fluid affecting the central subfield of the study eye at screening.
Exclusion criteria
Patients who have any contraindication and are not eligible for treatment with the chosen anti-vascular endothelial growth factor (anti-VEGF) treatment as according to the respective drug label.
Patients who have been on anti-VEGF treatment for longer than 3 years (before baseline visit).
Patients treated for retinal vein occlusion, diabetic macular oedema, myopic choroidal neovascularization (mCNV) or have diagnosis of diabetes-related macular degeneration within 6 months prior to the baseline visit.
For the complete listing, we refer to clinicaltrials.gov, Identifier NCT04662944.
Relevant data
Treatment-naive patients with nAMD to be treated or patients being treated for nAMD with aflibercept, brolucizumab, ranibizumab will be enrolled in the study on signing an informed consent. The baseline visit will be used to assess eligibility and collect baseline characteristics. De-identified patient’s data, including demographic data, nAMD characteristics, visit frequency, injection frequency and 3D OCT scans from Heidelberg Engineering, Topcon and Zeiss devices, will be collected (see figure 1, phase I). The study eye will be defined as the first eye to be treated during the study and the contralateral eye will be designated as the fellow eye. If both eyes are treated at baseline, the eye with the worse visual acuity will be chosen as the study eye. If the visual acuity is similar in both eyes, the treating ophthalmologist will designate the study eye at their discretion. The follow-up visits will take place at a frequency defined as per the site investigator. For RAZORBILL, patient data (i.e., age, medication, visual acuity etc) will be entered into an electronic case report form.
Patients who fail to receive an intravitreal anti-VEGF injection for at least 6 months or do not visit their ophthalmologist for at least 6 months will be discontinued from the observation.
OCT enrichment analysis (phase II)
AI segmentation of 3D OCT scans
The clinical data collected during phase I will be stored within the Discovery system (RetinAI Medical AG, Bern, Switzerland). Discovery is a cloud-based data management platform, which enables storing, viewing and processing imaging data, such as patient OCT scans.14–16 18
A set of advanced CE-marked AI algorithms, based on deep neural networks, is hosted in this platform, allowing for analysis and real-time enrichment of 3D OCT scans.14–16 A fully convolutional neural network with encoder–decoder architecture was trained on over 5000 B-scans of nAMD eyes for the segmentation and volumetric assessment of anatomical structures such as pathological fluid compartments.
The analysis (including but not limited to segmentation) will be performed in the following way:
Retinal layers are segmented and measured.
Pathological fluid compartments are identified, highlighted and quantified.
This volumetric segmentation will be overlaid on the standard ‘raw’ OCT image and describes precisely and quantitatively the retinal fluids, which is known to be the cornerstone of disease activity evaluation.11 19
The segmentation has been evaluated in a clinical study, which showed the dice similarity coefficient for intraretinal fluid (IRF), subraretinal fluid (SRF) and pigment epithelium detachment (PED) detection of 0.73, 0.67 and 0.82, respectively. The correlation coefficients for the fluid volumes between manual and automatic segmentation were 0.99, 0.99 and 0.91, respectively. For multiple acquisitions of patient on the same day, repeatability of volume prediction showed SD of 4.0 nL, 3.5 nL and 20.0 nL for IRF, SRF and PED, which amounted to 6.9%, 3.6% and 2.4% of the mean fluid volume, respectively.18 Exemplary outcomes of the same are illustrated in figure 2.
Automatic segmentation results (bottom) of exemplary original OCT scans (top) for IRF, SRF and PED, retinal layers and choroid. IRF, intraretinal fluid; OCT, optical coherence tomography; PED, pigment epithelium detachment; SRF, subretinal fluid.
Review of OCT scans (enriched vs non-enriched, original OCT)
On collection of clinical data (phase I), de-identified 3D OCT scans taken at patient study visits (along with 3D OCT scans from up to two earlier visits) will be shared with a panel of ophthalmologists for retrospective review (see figure 1, phase II).
The data will not be shared for at least 2 weeks to ensure that no treatment decision can be influenced by the review process in the phase II as treatment decision would have been made prior in phase I.
The ophthalmologists will be randomly divided into two groups (group A and B) and will independently review the same set of 3D OCT scans for disease activity, once with enhanced OCT information based on the AI applied algorithm and once without. These groups are allocated in a randomised manner via the Discovery platform. Each OCT will automatically be assigned by the platform for review to six ophthalmologists comprising of equal numbers of colleagues who participated in the data collection as well as general ophthalmologists who did not contribute to the data collection. However, physicians from the data collection phase will not be allowed to review OCTs of patients they treated in the data collection phase to prevent potential bias. For review, the ophthalmologists can freely navigate through the 3D OCTs and will enter its assessment results in Discovery. In the background, the tool will measure the time spend on the individual OCT assessments for analysis. All reviewing ophthalmologists will receive a standardised training on the OCT reviewing process prior to start of phase II.
After 8 weeks, in a crossover design fashion (see figure 1), physicians who review the enriched OCT scans in the initial analysis will review the original OCT (without enrichment) of the same data set when it is shared after 8 weeks. Conversely, physicians who review the original OCT in the initial analysis will be asked to review the enriched OCT scans after 8 weeks. This crossover is intended to eliminate bias and elevate the power of this investigation. The time delay of 8 weeks was chosen to prevent any notable carry over effect due to ‘memorisation’. Additionally, the order in which a reviewer receives images (enriched or non-enriched first) will be varied during the study to account for any residual remembering of the initial review. Furthermore, the same reviewers will not be consistently assigned to the same OCTs in order to allow reasonable overlap among all reviewers.
Study objectives
RAZORBILL is designed to assess the influence of automated OCT scan enrichment on disease activity assessment. The primary endpoint is the odds ratio of disease activity identification from OCTs with and without automated OCT enrichment. Two key secondary endpoints to further inform our objective are the degree of agreement in classification of disease activity using enriched OCT scans compared with using non-enriched OCT scans.
Further key objectives are summarised in box 2.
Key secondary objectives
Assess the influence of automated optical coherence tomography (OCT) segmentation on duration of OCT review.
Influence on confidence of disease activity assessment (questionnaire questions).
Assess if Discovery is accepted by physicians and whether it can optimise the ophthalmic clinical workflow (questionnaire questions).
For the complete listing, we refer to clinicaltrials.gov, Identifier NCT04662944.
Statistical analysis
To assess the influence of automated OCT scan enrichment with segmentation information on disease activity assessment, a generalised linear mixed model (GLMM) will be employed. The GLMM on disease activity (present/absent) will include fixed effects for enrichment status (yes/no), crossover period and a random effect for reviewer. An OR of disease activity identification from OCTs with and without automatic segmentation will be reported with a 95% CI.
Additionally, the degree of agreement in classification of disease activity across reviewers (with and without segmentation, separately) will be assessed by Krippendorff’s alpha. A bootstrap 95% CI for alpha will be reported.
Discussion
AI-based algorithms offer great opportunities in optimising patient treatment and will revolutionise today’s health care.20 Anatomical assessment of the retina with an OCT scan is currently the most central and essential tool for treatment decisions in patients with nAMD.11 Specific anatomical criteria allow for disease activity assessment, which recommends treatment or retreatment with intravitreal injection of an anti-VEGF agent. The experience and knowledge of the interpreting ophthalmologist influence the appropriateness of the treatment decision.21 Unfortunately, it has been seen in numerous real-world observational trials that significant undertreatment is common.4 In a well-controlled clinical trial, an average of approximately eight ranibizumab injections in the first year of treatment allows for the best treatment outcome measured with visual acuity.6 Global real-world data show that patients are treated about five times in the first treatment year,22 which also translates in lower long-term visual acuity outcomes when compared with randomised controlled trials. Besides misinterpretation of OCT scans, other reasons for the undertreatment may include the chronicity of the disease, patients’ non-adherence to retreatment schedules, lack of rigorous monitoring and poor compliance with recommendations of the treating physician.23 24
The RAZORBILL study investigates AI-based algorithms, which assist the ophthalmologist in the interpretation of OCT scans. The segmentation algorithms analyse 3D retinal OCT information to extract pathological features (including providing volumes of pathological fluid) and visualise these by highlighting the affected areas within the OCT scans. This volumetric information is potentially of high clinical value as it describes precisely and quantitatively the fluid status, which is known to be one of the cornerstones of disease activity evaluation.11 19 Where there is active disease, this may influence treatment decision with an intravitreal anti-VEGF agent and may ultimately improve disease control with better visual acuity outcome for the patients. With RAZORBILL, we aim to understand and evaluate the added value that AI-based enrichment of OCT images will bring to the task of disease activity assessment. Additionally, the study will provide essential information on the feasibility and applicability of the AI tool in the clinical practice setting.
Further evolution and advances in the field of AI applied to medical images will continue to improve algorithms, which will offer increased interpretation and even prediction capabilities. In the nAMD space, methods to perform automated diagnosis25 and treatment outcome estimation were recently reported.26–28
However, before novel AI tools can be applied in routine clinical care, investigations of their safety, their benefit as well as operational feasibility need to be carefully investigated.29
Conclusions
In this manuscript, we have discussed how RAZORBILL will assess the benefits that a set of segmentation algorithms applied to a 3D OCT scan data set collected via a routine clinical care will bring to the ophthalmology community. This innovative study design can also be readily applied to probe other newly emerging tools.
RAZORBILL is taking the first step and paving the way for the application of advanced AI algorithms to support treatment decisions in patients with nAMD in routine clinical care.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
Ethics statements
Patient consent for publication
Acknowledgments
The authors thank Dr. Sandro I. De Zanet, Dr. Stefanos Apostolopoulos and Dr. Carlos Ciller (RetinAI Medical AG) for providing support, scientific input and feedback on the RAZORBILL study and this manuscript. BG thanks Dr. Eser Adiguzel, Dr. Chiara Biancotto, Fiona Flynn Smith, Dr. Martina Junge, Dr. Christof Junkes, Nathalie San Nicolas, Dr. John Seaman, April Spollen, and Gabriela Susanna for fruitful discussions and input on the RAZORBILL study design.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors AC and BG contributed equally to this work. AC and BG designed the study with input from all co-authors. AC and BG drafted the manuscript. All authors reviewed and revised the manuscript.
Funding This work is funded by Novartis Pharma AG, Basel, Switzerland (no grant number existing).
Competing interests FGH reports grants and personal fees from Novartis, during the conduct of the study; grants and personal fees from Acucela, Allergan, Apellis, Bayer, Bioeq/Formycon, Geuder, Heidelberg Engineering, IvericBio, Kanghong, Roche, and Zeiss; personal fees from Grayburg Vision, Oxurion, and Pixium; grants from Biogen, and Optos, outside the submitted work. RAG reports personal fees from Allergan, Bausch & Lomb, Nidek, and Novartis, outside the submitted work. FB reports personal fees from Allergan, Bayer, Boehringer-Ingelheim, Fidia Sooft, Hofmann La Roche, Novartis, NTC Pharma, Sifi, Thrombogenics, and Zeiss, outside the submitted work. RD reports grants and personal fees from Bayer, and personal fees from Novartis, outside the submitted work. LO reports grants from Novartis, and personal fees from Bayer, outside the submitted work. DP has nothing to disclose. GS reports grants and personal fees from Centervue, Nidek and Novartis; grants from Carl Zeiss Meditec, Heidelberg Engineering, Ocular Instruments, Optos, Optovue, Quantel Medical, and Topcon; personal fees from Apellis, Allergan, Astellas, Bayer, Boheringer, Genentech, Graybug, Iveric, Roche, Chengdu Kanghong Biotechnology, Kyoto Drug Discovery & Development, outside the submitted work. AW reports grants from Novartis, outside the submitted work. DL reports full-time employment of Novartis Pharma AG. AC reports full-time employment and shareholder of Novartis Pharma AG. BG reports full-time employment at Novartis Pharma GmbH and is a shareholder of Novartis Pharma AG.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Highlights from this issue