Aims To determine the accuracy of diagnosing microbial keratitis by masked medical and non-medical observers using the Heidelberg Retina Tomograph II/Rostock Cornea Module in vivo confocal microscope.
Methods Confocal images were selected for 62 eyes with culture- or biopsy-proven infections. The cases comprised 26 Acanthamoeba, 12 fungus, three Microsporidia, two Nocardia and 19 bacterial infections (controls). The reference standard for comparison was a positive tissue diagnosis. These images were assessed on two separate occasions by four observers who were masked to the tissue diagnosis. Diagnostic accuracy indices, κ statistic and percentage agreement values were calculated. The Spearman correlation coefficient (rs) was calculated for the number of correct diagnoses versus duration of disease.
Results The highest sensitivity and specificity values were 55.8% and 84.2%, respectively, and the lowest sensitivity and specificity values were 27.9% and 42.1%, respectively. The highest positive and lowest negative likelihood ratios were 2.94 and 0.59, respectively. Agreement values were: fair to moderate (κ 0.22–0.44) for reference standard versus observer diagnosis, moderate to good in intraobserver variability (repeatability, κ 0.56–0.88) and poor to moderate in interobserver variability (reproducibility, κ 0.15–0.47). The correct diagnosis was associated with duration of disease for Acanthamoeba keratitis (rs=0.60, p=0.001).
Conclusions The diagnostic accuracy of microbial keratitis by confocal microscopy is dependent on observer experience. Intraobserver repeatability was better than interobserver reproducibility. Difficulty in distinguishing host cells from pathogenic organisms limits the value of confocal microscopy as a stand-alone tool in diagnosing microbial keratitis.
- Acanthamoeba keratitis
- confocal microscopy
- diagnostic accuracy
- diagnostic tests/investigation
- fungal keratitis
Statistics from Altmetric.com
- Acanthamoeba keratitis
- confocal microscopy
- diagnostic accuracy
- diagnostic tests/investigation
- fungal keratitis
Difficulties in clinical and microbiological diagnosis are one of the major problems in the management of microbial keratitis, particularly when caused by protozoa (Acanthamoeba and Microsporidia), fungi or filamentary bacteria. Diagnosis of these pathogens is difficult as they often take days or weeks to grow in culture and, in any case, culture is insensitive, with culture-positive rates rarely exceeding 60%.1 Although culture is still the primary diagnostic tool in tertiary referral centres it is not widely available to many patients because of limited resources.
The confocal microscope allows detailed in vivo analysis of normal2 and pathological corneas. In patients with presumed corneal infection, it is used in diagnosis and examination of the extent of involvement of tissue by infection and associated inflammation. All published studies have been directed at diagnosis and a number have shown both white light and laser confocal microscopy to be effective in diagnosing Acanthamoeba,3–5 fungal,6–8 Nocardia9 and Microsporidia keratitis.10 However, these studies only present case series or reports and there are limited published data on evaluating the diagnostic accuracy of confocal microscopy. Two recent studies have found high sensitivity and specificity values for diagnosing fungal keratitis (FK) and Acanthamoeba keratitis (AK) with the Confoscan 3.0 (Nidek Technology, Padova, Italy).11 12 However, factors such as observer or selection bias, the absence of masking the observers from the microbiological diagnosis, and lack of appropriate controls may have resulted in overestimates of the sensitivity and specificity values. Although experience in interpreting confocal keratitis images is essential, the accuracy of diagnosing microbial keratitis by clinicians with differing levels of confocal microscopy experience and the potential of using trained technicians in interpreting images have not previously been assessed. These are important considerations in evaluating this technique. The aim of this study was to examine the diagnostic accuracy of microbial keratitis with the Heidelberg Retina Tomograph II/Rostock Cornea Module (HRT II/RCM) in vivo confocal microscope, as a stand-alone tool, by trained medical and non-medical observers with differing confocal microscopy experience.
Materials and methods
This study was approved by the Moorfields Eye Hospital and Whittington Research Ethics Committee and it adhered to the tenets of the Declaration of Helsinki. We retrospectively reviewed the case notes of a consecutive series of patients with microbial keratitis who had had both corneal cultures or corneal biopsy and confocal microscopy (n=105) from 1 January 2005 to 4 January 2008. These cases were both those refractory to conventional treatment and those with unusual clinical features such as perineural infiltrates and ring infiltrates. Patients were referred either from Moorfields Emergency Department or from other institutions. Of the 105 cases, 62 culture- or biopsy-positive cases (62 eyes) were identified: 26 Acanthamoeba, 11 fungus, one fungus and bacteria, three Microsporidia, two Nocardia and 19 bacteria. Bacteria were used as controls because they are normally too small to detect with confocal microscopy9 13; therefore, the case which was culture positive for both fungus and bacteria was classified as an FK for the purposes of the study. We did not classify Nocardia as controls because they are filamentous bacteria and can form filamentous structures that are large enough to be distinguished by confocal microscopy.9 Empirical treatments started prior to assessment in this study included topical antimicrobial agents for presumed herpes, bacterial or keratitis of unknown cause, respectively. Irrespective of the referring diagnosis, all patients had undergone a full clinical examination by a corneal specialist and repeat corneal scraping for culture and confocal microscopy on the same day. If the scraping was culture negative, and the keratitis progressive, then a corneal biopsy was later performed. Exclusion criteria were culture- or biopsy-negative keratitis cases, and patients who declined to have confocal microscopy or a corneal culture as part of their clinical investigation. The reference standard for this study was either a diagnosis by isolation on culture of a corneal scraping or a histological diagnosis on a corneal biopsy; other ancillary culture sources such as contact lens case and solutions were not used. The clinical outcomes were recorded for all the patients in the study and were consistent with the diagnosis based on culture or histology; therefore, it is unlikely, but possible, that there were unrecognised polymicrobial infections which may have been identified on confocal microscopy but not by culture or biopsy. We followed the Standards for Reporting of Diagnostic accuracy (STARD) initiative in conducting this study.14
Culture and biopsy methods
Corneal scrapings for microbial culture were inoculated on the following media: blood agar, Sabouraud's dextrose agar (fungi), Robertson's cooked meat (anaerobic bacteria), Escherichia coli-seeded non-nutrient agar (Acanthamoeba), brain heart infusion (fastidious organisms, fungi) and Lowenstein–Jensen (mycobacteria, Nocardia). Scrapings were smeared on sterile glass slides for Gram and Giemsa stains. All microbiological investigations were undertaken independently in an external laboratory. For biopsy, a superficial lamellar disc of the affected cornea was trephined under local anaesthetic to provide a further specimen for microbiology and histopathological staining.
Confocal microscopy measurement protocol
In vivo confocal microscopy was performed on all 62 eyes by a single experienced observer (SH) with the HRT II/RCM (Heidelberg Engineering, Dossenheim, Germany) confocal microscope following a standard operating procedure as follows. A sterile Tomocap (Heidelberg Engineering) was mounted over the objective of the microscope (Zeiss, Jena, Germany; ×63), and polyacrylic acid 0.2% (Viscotears, Novartis, Camberley, UK) was used as a coupling agent between the cap and the lens objective. Topical anaesthetic (0.5% proxymetacaine hydrochloride, Bausch & Lomb, Kingston-upon-Thames, UK) and 1% Carmellose sodium (Celluvisc, Allergan, Marlow, UK) was instilled into both eyes to provide comfort and act as a coupling fluid between the front of the Tomocap and the cornea. Options for image acquisition include section (a single image at a particular depth), volume (a series of images over 60 μm depth) and sequence scans (a video sequence at a particular depth). The volume scan option was selected for image acquisition because it allowed the capture of a large number of images over a short space of time. The central region of the corneal ulcer or corneal infiltrate was scanned first followed by the top, left, bottom and right margin of the lesion. At each point, the epithelial layer of the affected area was scanned first and the focal plane of the microscope adjusted until the whole depth of the ulcer or infiltrate had been scanned. When there was more than one infiltrate, the same scanning sequence was repeated for each infiltrate. The wavelength of the laser employed in the HRT II/RCM is 670 nm and each standard two-dimensional image consists of 384×384 pixels covering an area of 400 μm×400 μm. The axial resolution is 7.6 μm; compared with other instruments such as the Tandem scanning microscope (9 μm) and Confoscan 4 (29 μm).15
The confocal images of all the scans were reviewed by two experienced confocal microscopists (SH and JD). In diagnosing keratitis, a considerable amount of time is often needed to find an image that would yield sufficient information to be able to identify the organism. This is due to masking of the organisms by the cellular inflammatory response and the fact that they seldom distribute evenly within the cornea during active infection. Therefore, to ensure all our observers had the maximum likelihood in diagnosing the type of keratitis, the best quality 384×384 pixel resolution digital image clearly indicating the culture-proven pathogen from the corneal ulcer or infiltrate was selected and exported onto Microsoft Power Point. These included those of Acanthamoeba—round single- or double-walled hyper-reflective objects (∼10–20 μm) consistent with Acanthamoeba cysts4 5; fungus—linear irregular branching hyper-reflective objects consistent with fungal hyphae6 7; Microsporidia—small round hyper-reflective deposits (∼2 μm) located in between keratocytes10; Nocardia—small branching filamentous structures within the corneal stroma9; and bacteria (control)—a mixture of inflammatory cells.
Intraobserver and interobserver agreement
All digital images were assessed prospectively in the same standard fashion in the Reading Centre at Moorfields Eye Hospital by four observers (three ophthalmologists and one medical technician) with differing levels of experience in assessing keratitis on confocal microscopy as follows. Of the three ophthalmologists, observer A had 6 years of experience in assessing microbial keratitis with confocal microscopy, observer B, 10 years of experience in confocal microscopy but not keratitis, and observer C, 6 months of experience in assessing keratitis with confocal microscopy. Observer D was a medical technician who had 2 years of experience in performing confocal microscopy using the HRT II/RCM and analysing keratitis images but with no experience in the clinical appearance and treatment of different types of keratitis. To ensure each observer was familiar with the image appearance of different cell types obtained from the HRT II/RCM confocal microscope, examples of both normal cellular morphology and the standard images of different pathogens were shown in a presentation before their assessment. In addition, a series of five recent articles on diagnosing keratitis with the HRT II/RCM4 5 7 9 10 were given to each observer to read 2 weeks prior to their scheduled assessment date.
The confocal images were viewed in random order and assigned an identification number from 1 to 62. To ensure that there was masking between observers, the order of viewing the images were randomised by computer before being assessed by the next observer on a different day. No clinical details regarding each case were made available to the observers. Each observer assessed the series of images in a masked fashion on slide show in Microsoft Powerpoint and recorded the diagnosis corresponding to one of the following categories: AK, FK, Microsporidia (MK), Nocardia (NK) or bacterial keratitis (BK). A reference sheet showing the range of sizes of resident and inflammatory cells including epithelium and macrophages, and pathogenic cells for example diameter of Acanthamoeba cysts was given to each observer for comparison during each assessment. Intraobserver variability (repeatability) was evaluated by asking each observer to reassess the images, randomised in a different order, 3 weeks later in the same standard fashion. Interobserver variability (reproducibility) was assessed by determining the level of agreement in diagnosis between observers. Readings of all the digital images were collected on a standard pro-forma and analysed.
Data analysis was performed with SPSS V14.0 (SPSS, Chicago, Illinois, USA). We calculated sensitivity, specificity, and positive and negative likelihood ratios (LRs) for both image sets for each observer. Positive LR predicts the probability of a positive test result in patients with disease compared with those who do not have the disease. Negative LR predicts the probability of a negative test in those who have the disease compared with those who do not. The level of agreement between the reference standard and different observers, and both intraobserver and interobserver variability were determined using the κ statistic. The interpretation of the κ statistic is as follows: ‘poor’ if κ ≤0.20, ‘fair’ if κ 0.21–0.40, ‘moderate’ if κ 0.41–0.60, ‘substantial’ if κ 0.61–0.80 and ‘good’ if κ >0.80.16 In addition, we also calculated percentage agreement values between reference standard and observers, within observers and between different observers. The Spearman rank correlation coefficient (rs) was used to determine the relationship between the number of correct diagnoses and the duration of disease for AK, FK and BK. The duration of disease was defined as the time from symptom onset to presentation to the Corneal and External Disease Service at Moorfields. A value of p<0.05 was deemed statistically significant. MK and NK were excluded from this analysis because the numbers were too small.
The reference standard consisted of 52 culture-positive cases from corneal scrapings and 10 histopathologically confirmed cases on corneal biopsy. Sensitivity, specificity and LR values for each observer are shown in table 1.
The highest sensitivity value obtained was 55.8% and the highest specificity value was 84.2%. We found fair to moderate agreement between observers and the reference standard (κ 0.22–0.44), moderate to good agreement in intraobserver variability (κ 0.56–0.88) and poor to moderate agreement in interobserver variability (κ 0.15–0.47), table 2. One observer (observer B) obtained the highest positive and lowest negative LR for diagnosing microbial keratitis. This observer also achieved the best overall κ and percentage agreement values in diagnoses compared with the reference, standard table 3. The best interobserver agreement (percentage agreement, 61.3–66.1%; κ 0.43–0.47) was between observer A and B, the two most experienced observers in the study. Observer C was the most repeatable (percentage agreement, 93.5%; κ 0.88) despite having the lowest κ and percentage agreement values compared with the reference standard (tables 2 and 3).
Complete agreement in diagnosis between all the observers and the reference standard for both assessments was found in 3/26 (11.5%) cases of AK, 8/19 (42.1%) cases of BK and 1/12 (8.3%) case of FK. In contrast, none of the observers identified Acanthamoeba in 5/26 (19.2%) cases or fungus in 4/12 (33.3%), and confused BK with other diagnoses in 2/19 (10.5%) cases. Observer B was the only one who managed to diagnose NK correctly in one case. The percentage correct diagnosis for the different types of keratitis is shown in table 4. A breakdown of all the diagnoses for each observer for the different keratitis category is shown in the appendix, available online.
Figure 1 shows a series of images demonstrating the difference in appearance between correctly diagnosed versus incorrectly diagnosed cases. Figure 1A and F show a case of late diagnosed AK versus early diagnosed AK; note the presence of inflammatory cells in the epithelium in early AK, making distinction between host cells and Acanthamoeba cysts and trophozoites difficult, whereas in delayed diagnosed AK, single cysts or clusters of cysts were seen in the stroma, with minimal host immune and resident cells seen. The incorrectly diagnosed cases (figures E-J) demonstrate the difficulties in distinguishing host cells from pathogenic organisms, and Nocardia (figure 1J) from FK because of their similarity in appearance on confocal microscopy.
Figure 2 shows a plot between the number of correct diagnoses for AK, BK and FK versus the duration of disease (days). The graph shows a moderate correlation between the number of correctly diagnosed cases and the duration of disease for AK (rs=0.60, p=0.001), but not for BK (rs=0.17, p=0.49) or FK (rs=−0.19, p=0.57). Therefore, the longer the duration of AK, the higher the likelihood that a correct diagnosis was made by the observers in grading the confocal images.
Acanthamoeba and fungus are uncommon causes of corneal infection for which early diagnosis is paramount because it yields better prognosis and reduces ocular morbidity.17 18 Although the current reference standard for diagnosing microbial keratitis is corneal culture, the sensitivity varies because of numerous factors.19
The HRT II/RCM in vivo confocal microscope has been shown to be useful in diagnosing a range of pathogens, but validation studies of this new technology are few. A recent review has reported the efficacy of diagnosing infectious keratitis with confocal microscopy to be inconclusive, with the possible exception of AK.20 Our results show moderate sensitivity and moderate to high specificity values in diagnosing microbial keratitis with the HRT II/RCM confocal microscope, whereas both Kanavi et al11 and Tu et al12 found very high sensitivity (>90%) in diagnosing AK and FK, respectively, with the Confoscan 3. Tu et al,12 using multitest referencing standards, reported that when there are both clinical characteristics and objective evidence of AK, the adjunctive use of confocal microscopy exhibited a sensitivity of 90.6% and specificity of 100%. In our study, we set out to evaluate the diagnostic accuracy of confocal microscopy as a stand-alone tool rather than a supportive investigative technique, without the bias and influence of clinical findings. Although assessing confocal images in the absence of clinical data does not reflect the use of confocal microscopy in clinical settings, it is the only way to avoid bias when analysing the images. Our inclusion criteria were based on culture-positive cases irrespective of confocal classification. Although we chose only one representative image from each case, this was the best available image for the organism that was cultured from each case, giving the observers the best opportunity to make a correct confocal diagnosis; we believe that reviewing a series of images from each case would either have made a correct confocal diagnosis more difficult or have had no effect on the outcome. In addition, it allowed standardisation when viewing the images so that all observers assessed the same number of images consecutively. The absence of controls in the previous studies and the use of confocal-‘positive’ without culture confirmation as a reference standard, or for the case definition,11 12 could lead to selection bias and misdiagnosis, resulting in an overestimation of sensitivity values.12 21 22 This is evident from our controls in which immune cells can often be confused with AK cysts, and vice versa, leading to erroneous diagnosis. Furthermore, ‘good’ confocal images have been illustrated in most published studies to present findings without discussion of difficulties in analysing equivocal images. We found fair to moderate agreement between reference standard and observer diagnosis when a case mix of equivocal and unequivocal images was analysed by our observers. The rigorous criteria in our study design in regard to the use of masked observers and controls could explain why sensitivity values, even for the most experienced observer, were lower.
Another explanation for the very high sensitivity values reported in one previous study was the use of only one ophthalmology-trained observer who, in addition to being unmasked to the clinical findings, was experienced in the use of confocal microscopy for keratitis diagnosis: this makes it difficult to extrapolate the results to what might be expected outside their centres.12 To evaluate the potential of using this technology in clinics where an ophthalmologist with experience in confocal microscopy may not be available, our graders included two experienced ophthalmologists, an inexperienced ophthalmologist and an experienced technician. We found a twofold difference in sensitivity between the most experienced and the least experienced observer, indicating higher diagnostic accuracy with clinicians experienced in confocal microscopy. Our results indicate that the sensitivity value with a trained technician, with no experience in the clinical appearance of different types of microbial keratitis, was better than with an inexperienced medical observer, but with a lower specificity value and positive LR. This raises the possibility of training non-medical personnel in performing and analysing keratitis images. The highest positive LR and lowest negative LR was achieved by observer B who was experienced in confocal imaging of normal corneal anatomy and various pathological conditions other than microbial keratitis, indicating that experience gained in other aspects of confocal microscopy improves the diagnostic outcome.
Intraobserver agreement (repeatability) was found to be moderate to good, indicative of good observer repeatability in grading the images irrespective of the accuracy of their diagnoses. Observer experience did not appear to improve intraobserver repeatability as the observer with the lowest sensitivity had the highest repeatability, and vice versa. Interobserver agreement (reproducibility) was poor to moderate between different observers because of factors such as observer experience and differences in techniques of classifying images by different observers. The two observers who had the highest sensitivity values also had the best interobserver reproducibility, indicating that experienced observers achieved a higher diagnostic accuracy and reproducibility than less experienced observers. Therefore, to improve reliability the same experienced operator should be employed if sequential imaging of a patient is required.
Our observers were able to diagnose AK more accurately than any other type of keratitis. The unique appearance of Acanthamoeba cysts on confocal microscopy and the higher number of cases of AK compared with other conditions in our study might explain this outcome. However, AK was commonly confused with controls, and vice versa, because of the diagnostic difficulty with some of the equivocal images. There was a marked association between the accuracy of diagnosing AK and the duration of disease. Previous case reports have mainly described the morphological features of cysts and trophozoites in the epithelium and stroma during active infection,4 5 but have not related the number of cysts seen and the way they distribute with the different stages of the disease process. In early disease, where the organism is mainly confined to the epithelium, the presence of large numbers of inflammatory cells made diagnosing AK more difficult because of the difficulty in distinguishing AK cysts and particularly trophozoites from inflammatory cells.5 Late presentation was associated with either a greater number of Acanthamoeba cysts seen in the images or the fact that they were easier to identify because of a reduction in the type and number of host cells seen. Our experience, therefore, suggests that AK is easier to identify with confocal microscopy in the later stages of infection.
The use of confocal microscopy in diagnosing FK has been widely reported in the literature.6–8 Filamentous fungal hyphae have characteristic linear hyper-reflective lesions branching at a 45 or 90° angle,7 whereas Candida infection produces pseudofilaments.7 Despite these well described confocal appearances of FK in the literature, the percentage of correct diagnoses in our series was low, possibly due to difficulties in differentiating other linear images from fungal hyphae.23
Nocardia and Microsporidia species are rare causes of microbial keratitis.24 Clinically, Nocardia may be misdiagnosed as mycotic or mycobacterial keratitis,9 25 while Microsporidia can be misdiagnosed as AK or herpes simplex keratitis. Despite the rarity of these organisms, because of the unique appearance on confocal microscopy with Microsporidia,10 two observers managed to identify this organism correctly in both of their assessments. Only observer B managed to obtain the correct diagnosis in both assessments for diagnosing one case of NK; the unfamiliarity in interpreting confocal images of Nocardia, the similarity in appearance of fungal hyphae and Nocardia filaments, and the small number of cases in our study made diagnosing this organism difficult. The inclusion of both Nocardia and Microsporidia cases might have reduced the overall sensitivity and specificity values but, as confocal findings of both organisms have been reported, we believe it was appropriate to include them in the study.
In summary, to the best of our knowledge, this is the first study evaluating the diagnostic accuracy of microbial keratitis using a single reference standard for different masked observers with the HRT II/RCM confocal microscope. Although confocal microscopy is non-invasive and can provide a rapid diagnosis for microbial keratitis, (1) similarities between inflammatory and pathogenic cells and (2) difficulty in interpreting equivocal images limits its usefulness as a stand-alone tool in diagnosing keratitis. Confocal microscopy is a useful adjunct in managing refractory cases and we have shown that the diagnostic accuracy improves with clinician experience. However, the diagnostic accuracy of confocal microscopy used in isolation from the clinical assessment is still too low to be a substitute for tissue diagnosis, particularly in patients with progressive disease. Improvement in clinician training and experience, greater standardisation of image interpretation and the development of new software in tandem with higher resolution imaging is likely to improve the diagnostic accuracy of this technology in diagnosing microbial keratitis in the future.
The authors would like to thank Dr Catey Bunce for her statistical advice and support.
Linked article 182550
Competing interests None.
Ethics approval This study was conducted with the approval of the Moorfields and Whittington Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.