Article Text
PDF
Abstract
Background/aims Digital retinal imaging using store-and-forward technology is used to screen for diabetic retinopathy (DR). Its usefulness in detecting non-diabetic eye diseases is uncertain. We determined the level of agreement between teleretinal imaging supplemented with visual acuity and intraocular pressure (IOP) measurements (ie, technology-assisted eye (TAE) exam) and a comprehensive eye exam in evaluation for DR and non-diabetic ocular conditions.
Methods We conducted a prospective, observational study with two parallel evaluations. Patients with diabetes (n=317) had a TAE exam and a comprehensive eye exam on the same day. A subset of participants with normal baseline exams (n=72) had follow-up exams 1 year later. We measured the level of agreement for referable ocular findings.
Results Agreement for referable ocular findings was moderate (n=389, agreement: 77%; κ: 0.55), due in part to ungradable exams (22%). However, about half of the ungradable exams had findings that warranted referral. There was substantial agreement for follow-up exams (n=72, agreement: 93%; κ: 0.63). Among all gradable exams (n=303), the TAE exam had 86% sensitivity and 84% specificity for referable ocular findings, with high agreement (≥94%) for DR and other major ocular diagnoses.
Conclusions There was moderate-to-substantial agreement between a TAE exam and a comprehensive eye exam for referable ocular findings in patients with diabetes. Ungradable exams were a frequent marker of ocular pathology. Teleretinal imaging may be a useful evaluation for both diabetic and non-diabetic ocular conditions.
- Telemedicine
- Imaging
Statistics from Altmetric.com
Introduction
Screening and treating patients for diabetic retinopathy (DR) reduces severe vision loss.1 While the incidence of vision-threatening DR is low, patients still require ongoing surveillance. Digital retinal imaging coupled with store-and-forward technology (ie, teleretinal imaging) may be used as a screening tool given its ability to capture diagnostic quality images that can be interpreted remotely. Teleretinal imaging has been successfully incorporated into the eye care of patients, particularly in primary care settings.2–8
While teleretinal imaging has been validated for DR screening, its ability to detect concomitant non-diabetic eye diseases is less certain.8 ,9 As the technology is deployed broadly for DR screening, it remains uncertain if retinal imaging per se can replace periodic complete eye exams to detect diabetic and non-diabetic eye diseases. This is particularly important in low-risk patients where technology-based screening is likely to be more cost effective.10 ,11 Thus, it is important to know the accuracy of an imaging-based eye exam in detecting DR and other forms of referable eye disease in comparison with a comprehensive eye exam.
We hypothesised that teleretinal imaging supplemented with a non-invasive eye care assessment (hereafter referred to as a technology-assisted eye (TAE) exam) provides clinical information approaching that obtained from a comprehensive eye exam. This study was conducted to assess the potential role for a TAE exam in the routine eye care of patients with diabetes.
Subjects and methods
Study design
The study employed a prospective, observational design to compare the level of agreement between a TAE exam and a comprehensive eye examination in patients with diabetes. The study was reviewed and approved by the Human Subjects Committee of the Department of Veterans Affairs (VA) Boston Healthcare System and written, informed consent was obtained from all participants.
Patients with diabetes mellitus between ages 18 and 80 years were enrolled. Study participants were recruited from the Primary Care and Eye clinics (Optometry and Ophthalmology) at VA Boston Healthcare System. Lists of scheduled patients were reviewed in advance for eligibility criteria. Potential study participants were sent a recruitment letter or approached face to face when they presented for a scheduled eye care appointment. All patients completed both a TAE exam and a comprehensive eye exam on the same day in separate examination rooms. Independent examiners completed each exam and were masked to the results of the alternate evaluation. The order of exams was fixed: the TAE exam was completed first as it was conducted without pupil dilation. Examiners had access to the patients’ electronic health record and could apply known ocular findings and previous diagnoses in their final assessments.
TAE exam
The TAE exam was performed by a trained technician, certified by the Veterans Health Administration Office of Telehealth Services, National Store and Forward Telehealth Training Center (SFTNTTC) in the acquisition of non-mydriatic fundus images. Technicians were also trained to obtain the elements of the TAE exam which included a brief medical and ocular history, measurement of visual acuity and intraocular pressure (IOP) using non-contact tonometry (NIDEK, Fremont, California, USA). Visual acuity was measured with the patients’ corrective lenses and/or pinhole using a projected Snellen eye chart and was recorded as a fraction and converted to logarithm of the minimum angle of resolution (logMAR). Two IOP readings were obtained for each eye; if there was ≥5 mm Hg difference between readings, a third measurement was obtained. Teleretinal imaging employed a validated protocol2 ,5 ,7 using a non-mydriatic fundus camera (Topcon TRC NW-6S, Paramus, New Jersey, USA) to acquire a minimum of eight single-frame non-stereoscopic images. The validated imaging protocol included a single external image of each eye and three non-stereo 45° field images including Field NM-1, a macular-centred central image that included the optic disc, Field NM-2 superior temporal to the optic disc and Field NM-3, nasal and slightly inferior to the optic disc (see figure 1). Retinal images and associated clinical data were reviewed asynchronously by one of three optometrists who were trained in the standardised reading of teleretinal images according to protocol and certified as teleretinal readers/graders by the SFTNTTC. Results of the exam included the presence or absence of any concomitant referable eye findings and level of DR and/or diabetic macular oedema.
An example of the three standard 45° retinal fields captured by teleretinal imaging from a patient graded with normal findings: (A) central, (B) superior and (C) nasal fields. (D) External view.
Comprehensive eye exam
The comprehensive eye exams were performed by one of two optometrists or a board-certified ophthalmologist and employed a typical clinical protocol including best corrected visual acuity with refraction, IOP measured by Goldmann applanation tonometry, confrontation visual fields, external/adnexal examination, a slit lamp biomicroscopic exam and a dilated fundus examination by indirect ophthalmoscopy. Additional tests, including gonioscopy and scleral indentation, were performed when indicated at the discretion of the examiner. The examiners who performed the comprehensive eye exams did not evaluate any of the TAE data, including the images and vice versa.
Subgroups
To assess the stability and reliability of comprehensive eye exam findings, the first 128 patients were invited to return within 4 weeks of their baseline exam for a second complete eye exam by an independent optometrist or ophthalmologist; 45 agreed to participate. This subgroup consisted of 37 patients with normal baseline findings and 8 patients with abnormal baseline findings.
To assess the incidence of new findings and agreement between exams over time, we invited all patients with normal baseline findings (n=124) to return for a follow-up eye exam in 1 year. Of those invited, 72 patients (58%) returned for follow-up study exams.
Outcome measures and statistical analyses
We considered any ocular finding that warranted eye care follow-up within 1 year as a referral recommendation. Readers graded DR and diabetic macular oedema using the International Clinical Disease Severity Grading Scale.12 A DR-related referral recommendation was made for findings greater than mild non-proliferative DR or suspicion of macular oedema. Non-diabetes-related referrals were for visual acuity ≤0.5 (≥0.30 logMAR or Snellen ≤20/40), glaucoma suspicion, cataract and age-related macular degeneration (AMD). Referrals for glaucoma or glaucoma suspicion were based on any of the following findings: (1) pre-existing glaucoma diagnosis, current anti-glaucoma therapy or known risk factors (eg, ocular hypertension), (2) IOP asymmetry ≥5 mm Hg, (3) IOP ≥25 mm Hg in either eye, (4) cup disc ratio (CDR) ≥0.6 in either eye, (5) CDR asymmetry ≥0.2 in either eye, (6) disc haemorrhage in either eye or (7) nerve fibre layer defect in either eye. Referrals for AMD were based on the presence of Age-Related Eye Disease Study category 3 or higher macular changes. Less common ocular pathologies were diagnosed using standard clinical criteria. If the reader could not grade all components of the TAE exam, the patient received a referral recommendation.
The primary outcome measure was referral recommendation for follow-up eye care in less than 1 year. Other outcomes of interest were agreement for individual ocular findings. Since each exam instance (baseline and follow-up) served as an independent evaluation, we determined level of agreement for each exam period and for the combined baseline and follow-up exams. To assess accuracy of ocular findings, estimates of sensitivity, specificity and predictive value for the TAE were calculated against findings from the comprehensive eye exam, which in this case was considered the ‘gold standard’. Level of agreement was determined by both percentage agreement and κ statistic for inter-rater agreement. Between-group comparisons were performed using a t test. All analyses were conducted using SAS V.9.2.
Results
Participant characteristics
We enrolled 317 participants with baseline study exams; 72 participants with a normal baseline exam also had a follow-up evaluation 1 year later. Characteristics of the baseline study population are shown in table 1. At least one portion of the baseline TAE exam was rendered ungradable in 81 individuals (26%). Of these, 31 (38%) were due to cataract, 19 (23%) were due to pupillary miosis and 31 (37%) were due to both cataract and miosis. Individuals with ungradable exams were older, and had longer duration of diabetes, lower IOP and lower visual acuity than those with gradable exams (table 2).
Study participant characteristics
Participant characteristics based on gradable baseline exams
Level of agreement between TAE and comprehensive eye exams
The two exams had moderate-to-substantial agreement for overall referral recommendations, with moderate agreement (74%; κ: 0.5) for the 317 baseline exams and substantial agreement (93%; κ: 0.63) among the 72 follow-up exams. Level of agreement between the two exams was affected by 86 TAE exams that could not be graded at baseline (n=81) or follow-up (n=5). These generated automatic referral recommendations. However, 44 of these 86 patients (51%) had referable findings identified during the comprehensive eye exam. When baseline and follow-up exams were pooled together (n=389), there was moderate agreement (77%; κ: 0.55) between the TAE and comprehensive eye exams for referable ocular findings (table 3).
Subject disposition for complete eye exam and TAE exam—all participants (n=389)
We next evaluated only those gradable TAE exams (n=303). There was substantial agreement (84%; κ: 0.67) for referral recommendations with the comprehensive eye exams (table 4). There was also a high level of agreement for DR (97%; κ: 0.68) and for individual ocular diagnoses, ranging from 84% to 99% (κ: 0.36–0.80). Among non-diabetes-related findings, cataract had the highest agreement and concomitant ocular diseases (a combination of less common ocular pathologies such as lid lesion, choroidal naevus, retinal embolic disease and anterior pupillary defect) had the lowest agreement. Suspicion or presence of glaucoma was the most common finding (23%) with both exams, but the prevalence of concomitant ocular diseases was relatively low.
Subject disposition for complete eye exam and TAE exam—all gradable exams (n=303)
Incidence of new referable eye exam findings
Among 72 patients with normal baseline exams, the incidence of new concomitant ocular diseases after 1 year was very low. Six patients (8%) developed referable eye findings as determined by the comprehensive eye exam, including cataract (n=2), DR (n=1), glaucoma (n=1) and other referable findings (n=2). The TAE and comprehensive exams had substantial agreement for the overall referral recommendation (93%; κ: 0.63) and DR (92%; κ: 0.66). Five studies (7%) were rendered ungradable at the follow-up exam, two due to cataract, one due to pupillary miosis and two due to both cataract and miosis.
Sensitivity and specificity
Among all baseline and follow-up exams (n=389), the referral recommendation from the TAE exam had 90% sensitivity and 69% specificity with 64% positive predictive value (PPV) and 92% negative predictive value (NPV). When only gradable exams were considered, the TAE exam had 86% sensitivity, 84% specificity, 73% PPV and 92% NPV. Results for individual diagnostic categories are shown in table 5.
Sensitivity, specificity, predictive value of technology-assisted eye exam referrals
Information yield from visual acuity and IOP measurements
We assessed if visual acuity or IOP measurements added to the referral recommendations. Four patients had TAE referrals for reduced visual acuity; two of these referrals were confirmed by the comprehensive eye exam and two were false-positive. No TAE exams had false-negative findings for reduced visual acuity. Four patients had elevated IOP measurements in the TAE exam that were not confirmed in the comprehensive eye exam. However, each of these patients had concomitant ocular diseases that warranted eye care referral. Two TAE exams had false-negative IOP referral recommendations. In these instances, the highest IOP recorded at the TAE exam was 17 and 24 mm Hg, respectively, whereas the complete eye exam recorded readings of 25 and 26 mm Hg, respectively.
Reproducibility of comprehensive eye exam findings
The comprehensive eye exam served as the gold standard with which the TAE exam was compared, but these exams also had some variation. Of 45 participants who had two comprehensive eye exams performed within 1 month, there was substantial agreement between the two exams (93%;κ: 0.73). All patients with normal findings at the first exam (n=37) had a normal second exam. Of eight patients with referable findings at the initial exam, only 5 (63%) were confirmed at the second exam. Findings were not confirmed in two cases initially labelled as glaucoma suspect and one case with DR.
Discussion
Teleretinal imaging has been successfully deployed in evaluating for DR in patients with diabetes. Over a range of populations, using different imaging systems and clinical pathways, teleretinal imaging has been shown to have high reliability to detect DR.2–7 ,13 ,14 Our findings support this role and also show that teleretinal imaging may also detect non-diabetic ocular findings with moderate-to-high reliability. We found substantial agreement for entities such as cataract, glaucoma and macular degeneration and less agreement for more rare findings. Measuring visual acuity and IOP did not substantially improve the frequency or accuracy of eye care referrals.
Level of agreement between the TAE and comprehensive eye exams was reduced by a number of ungradable exams, which was higher than typically found in teleretinal imaging programmes.5 ,15 This, in part, relates to our use of non-mydriatic imaging and the requirement that TAE exams be interpreted for the presence of both DR and other concomitant ocular diseases. Any exam feature that could not be graded rendered the entire study ungradable. This likely excluded some exams that still carried relevant clinical information. However, more than half of the ungradable patients had referable findings noted during the comprehensive eye exam. Thus, ungradable TAE exams may be a surrogate for referable ocular findings.
There are no generally accepted criteria for assessing accuracy of screening programmes to evaluate vision-threatening eye conditions per se. Nonetheless, we observed levels of sensitivity and specificity that were similar to or greater than screening programmes focused on DR detection alone16 ,17 or sight-threatening diseases that are diabetes related.18 Our results also exceeded UK standards for DR detection that require at least 95% specificity.19
This study has several strengths. We examined a large and diverse group of patients with diabetes with masked, side-by-side exams. Participants had both study exams on the same day to limit variations in findings. We validated the reproducibility of the comprehensive eye exam and showed perfect agreement when the exam was normal, but some variability when other findings were present. We also examined a cohort of patients 1 year later and validated the ability of TAE exams to identify incident findings. Nonetheless, there are some limitations of the present study. Our results may not generalise to other high-risk populations with greater proportions of minorities and women. Participants in our study had a low prevalence of non-diabetic ocular diseases with the exception of glaucoma and this makes the TAE exam appear less sensitive for some ocular findings. A larger sample would be needed to assess true concordance of such ocular conditions. Examiners had access to patients’ electronic medical records and some referral recommendations may have been influenced by the presence of known eye diseases. However, this replicates real-world practice whereby clinicians act on exam findings in the context of what is known about the patient's clinical condition. There were two cases of referable DR and elevated IOP missed by the TAE exam. It is difficult to know if this may have resulted in adverse outcomes. Lastly, non-mydriatic retinal imaging, while convenient for patients, may limit the ability to fully assess the retina and other intraocular structures due to small pupils or media opacities.
Many eye diseases including DR are asymptomatic in their early stages and thus dictate the need for periodic screening. Several studies and guidelines have confirmed that patients with diabetes with no DR and/or low-risk characteristics may have eye exams every 2–3 years.20–22 Even so, many obstacles prevent timely and appropriate eye screening including limited access and misperceptions by patients of the value of regular eye exams.23 ,24 Teleretinal imaging is both a convenient and cost-effective way to improve patients’ access to eye care.10 ,11 Indeed, primary care clinics are appropriate sites to engage patients at-risk for visual disability.3 ,25 A major advantage of teleretinal imaging is its scalability and applicability in numerous settings worldwide. Technicians or assistants can be trained as imagers and studies can then be sent to a reading centre where specially trained clinicians anywhere in the world can review and evaluate the studies. Multiple imaging sites can transmit images to the same reading centre, reducing administrative and operational costs and improving efficiency. Informed by our findings, teleretinal imaging may be useful for ongoing surveillance in place of routine comprehensive eye exams in patients with an initial normal exam.
In summary, we found moderate-to-high agreement for overall eye care referral recommendations and individual ocular diagnoses between gradable TAE exams and comprehensive eye exams. Therefore, teleretinal imaging may be a useful tool in evaluation of both diabetic and non-diabetic ocular conditions.
References
Footnotes
Funding This study was supported by research grants IIR 04-045 from the Veterans Affairs Health Services Research & Development Service and the National Institutes of Health (K24 DK63214).
Disclaimer The views expressed in this article are those of the authors and do not necessarily reflect the position and policy of the US Federal Government or the Department of Veterans Affairs. No official endorsement should be inferred.
Contributors PRC had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analyses. He researched data and wrote the manuscript. BA, LRP, GS and AAC researched data and reviewed/edited the manuscript, RL researched data and contributed to discussion.
Competing interests None declared.
Ethics approval VA Boston Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- At a glance