Article Text
PDF
Abstract
Background Despite high-quality evidence being essential for planning and delivering eye health programmes, evidence on what works is relatively scarce. To address this need, we developed eye health Evidence Gap Maps (EGMs) with the first one focusing on cataract. These maps summarise, critically appraise and present evidence in a user-friendly format. This paper presents experiences of developing the cataract gap map and discusses the challenges and benefits of the process.
Methods Following a comprehensive search of relevant databases, we sifted and extracted data from all relevant reviews on cataract. Critical appraisal was conducted by two reviewers independently using Supported the Use of Research Evidence checklist and a summary quality assessment was shared with the authors for comments.
Results A total of 52 reviews were included in the map. The majority of the reviews addressed quality of clinical care (20) and types of treatment (18). Overall, 30 reviews provided strong evidence in response to the research question, 14 reviews showed weak or no evidence and in 14 reviews the results were inconclusive. 14 reviews were regarded as high quality, 12 were medium quality and 26 were graded as low quality. To verify the validity of the Supporting the Use for Research Evidence (SURE) checklist, studies were also appraised using the Scottish Intercollegiate Guidelines Network (SIGN) tool. Based on the κ statistics test, results showed excellent agreement between the two checklists (K=0.79).
Discussion EGMs support policy makers and programme managers to make informed decisions and enable researchers to prioritise future work based on the most evident gaps on knowledge.
- Eye (Globe)
- Field of vision
- Vision
- Epidemiology
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Background
Visual impairment
Visual impairment is a global health problem, with an estimated 233 million people affected including 39 million who are blind.1 ,2 Around 65% of blindness and 76% of moderate and severe visual impairments (MSVIs) can be avoided through timely access to prevention or treatment, but access to such interventions in many low/middle-income countries (LMICs) continues to be limited.1
Cataract is responsible for 33% of blindness and 18% of MSVIs globally, and a lack of access to good quality eye health services is a key determinant of avoidable vision loss due to cataract.3 Studies among people with unoperated cataracts in sub-Saharan Africa and Asia highlighted constraints affecting both supply of (eg, number of qualified health workers) and demand for services (eg, access to services).4–9 Rigorous and relevant research evidence is essential for planning cost-effective and scalable approaches to deal with avoidable visual impairment in resource-constrained environments. Evidence on interventions that could be used to improve eye health in LMICs is scarce, and there is no single repository where all relevant research may be found in a user-friendly format. Many policy makers and clinicians plan their interventions without the benefit of the most up-to-date knowledge, and patients may not be receiving the best possible care available. Without a comprehensive overview of existing evidence, current research efforts may not be focusing on the most pressing priorities with the most urgent evidence gaps.
Evidence gap maps
Evidence gap maps (EGMs) are a visual tool for presenting the state of evidence in particular thematic areas relevant to international development, with the aim to provide easy access to the best available evidence and highlight knowledge gaps. The approach was developed by the International Initiative for Impact Evaluation (3ie) and has been applied to a number of development areas including water and education.10 ,11 Fundamentally, the maps are matrices with interventions plotted against outcomes. The evidence populating an EGM can be drawn from any source, but many existing EGMs synthesise evidence from systematic and literature reviews. All evidence is assessed on its relevance to LMICs, and the quality of the review is appraised using a standardised quality appraisal tool. All sources of evidence meeting predefined inclusion criteria are plotted in the map cells corresponding to specific interventions. The cells are linked to summary pages and show methodological quality of the review using a traffic light system. A glance at the map clearly shows where the evidence lies and where little evidence or poor quality evidence exists. This paper describes experiences and results of constructing the first EGM in eye health, which focused on cataract. The EGM was developed by the international non-governmental organisation (NGO) Sightsavers with contributions from academic centres working in the area of visual impairment, including Cochrane Eyes and Vision group. The map is available online and can be found in http://www.sightsavers.org/gap-maps/cataract-gap-map/.
Methods
The process of constructing the cataract EGM is shown in figure 1. As several reviewers were involved in the map development, the key was to ensure consistency throughout the process. Each author was assigned specific tasks for each stage of the EGM development; additionally, the main author oversaw and was involved in each activity. To avoid selection bias, two authors independently screened the identified studies against the predefined inclusion criteria. Two reviewers independently extracted data and followed a structured open-ended heading alongside guidance on the content to extract from each review. Critical appraisal was conducted using a standardised checklist and guidelines for appraising.
Overview of the cataract evidence gap map (EGM) development.
Search strategies
The search strategy aimed to identify all published systematic and literature reviews, including primary studies. To identify relevant sources, we systematically searched several databases including Cochrane Library, MEDLINE and EMBASE from inception to the end of 2013 (see online supplementary appendix A).
Supplementary appendices
Additionally, we reviewed reference lists of the included reviews and contacted the authors for information on other relevant sources.
Sifting and selection
Titles and abstracts were reviewed independently by two reviewers (from a team of two) against the predefined selection criteria:
Systematic or literature reviews
Reviews containing evidence from LMICs
Reviews published between 1993 and 2013
Reviews describing methods used for data collection and synthesis
Relevant to LMICs
The relevance was assessed by an ophthalmologist with substantial clinical experience in LMICs and a Sightsavers researcher. Reviews were excluded if the search methodology was restricted to high-income settings only, studies were narrative reviews, published before 1993, or did not describe their search and data synthesis methods.
Data extraction and strength of evidence
Two reviewers (from a team of seven) independently extracted data of studies meeting the inclusion criteria and assessed methodological quality of the reviews. A short summary describing the key findings of each review and key points around the methodology was developed.
Two researchers (from a team of two) independently analysed the strength of evidence based on the findings and conclusions reported by the authors. Evidence was categorised as strong if the review found consistent strong evidence in response to the research question or outcome; inconclusive if the review reported mixed results; and weak if the review found weak or no evidence in response to the research question or outcome. Disagreement between the reviewers was resolved by discussion. All data extraction and critical appraisal were in the original language of the review. For reviews written in other languages, reviewers were required to extract data and appraise it in English. To ensure consistency, the main author conducted a third review.
Quality appraisal
The methodological appraisal used an adapted version of the Supporting the Use for Research Evidence (SURE) tool developed by the SURE collaboration (see online supplementary appendix B).10 This checklist was used for all EGMs supported by 3ie. The tool gives reviews an overall rating of high, medium or low confidence based on the assessment of three main components : (1) methods used to identify, include and critically appraise studies; (2) methods used to analyse the findings and (3) reliability of the review.
Based on these, a summary of the methodological quality assessment was produced reporting the overall confidence attributed to the conclusions of the review. High confidence was attributed if all review methods were systematic and appropriate, as specified in the appraisal checklist, and the risk of bias was minimised. Medium confidence was attributed if all or some of the review methods were not systematic, as specified in the appraisal checklist, but the methodological limitations were acknowledged and taken into account in the conclusions of the review. Low confidence was attributed if overall review methods were not systematic or appropriate, and the methodological limitations were not acknowledged or taken into account in the conclusions of the review. All appraisals were conducted by two reviewers independently; if discrepancies were identified, a third reviewer independently extracted data and appraised the review.
Authors of all included studies were contacted and provided with the review quality appraisals for verification. If the authors were able to provide additional information to support a change in the quality assessment, this was sent to the original reviewers who again evaluated the paper independently.
To verify the validity of the SURE checklist for the purpose of this assessment, we applied another critical appraisal tool (the Scottish Intercollegiate Guidelines Network (SIGN) checklist) to the English and Spanish language studies, compared the results of the two appraisals and calculated the level of agreement between the tools using the κ statistic test (K).12 Compared with other critical appraisal tools, SIGN is the only one that assesses the overall methodological quality of a paper and is therefore comparable with the SURE checklist.
Scope and presentation of the cataract EGM
3ie EGMs visual presentation consists of interventions (y-axis) against outcomes (x-axis). We modified this structure to ensure that the cataract EGM clearly shows the strength of evidence available and plotted the thematic areas assessed in the reviews (x-axis) against the reviews' strength of evidence (strong evidence, weak evidence or mixed). We think this structure is more appropriate for the EGMs, which display evidence on medical-related thematic areas.
The way of organising the thematic areas reported in the selected reviews was determined through a consultation between the authors of the paper and two ophthalmologists with substantial experience in eye health research in LMICs. As a result, 14 thematic areas related to epidemiology and impact of cataract, clinical care and health systems were agreed. The term thematic area is used throughout the paper, as not all included reviews reported an outcome measure or intervention.
Each review meeting the inclusion criteria was placed in the cells corresponding to the relevant thematic area along the x-axis and the strength of evidence reported in the conclusions along the y-axis. Visually, each review is represented by a single bubble, which links to the review summary page (http://www.sightsavers.org/gap-maps/cataract-gap-map/). To enable users to visually identify the quality of each review, a traffic light system was used. Green, orange and red bubbles represent high, medium and low levels of confidence in review conclusions, respectively.
Results
Of the 1197 unique studies identified, 52 systematic or literature reviews were included in the cataract EGM (figure 2). Of the included studies, six related to two interventions and were included in the map twice, resulting in a total of 58 sources depicted on the map.
Documents identified through the search (from 29 July to 26 September 2013).
The identified reviews were mostly written in English, three were in Chinese, two were in German and two were in Spanish.
The majority of the reviews addressed quality of clinical care (20) and types of treatment (18). Of the remainder, nine looked at risk factors/prevention, five were economic evaluations and three each addressed the burden of the disease and the accessibility of cataract services (figure 3).
Thematic areas addressed by reviews included in the cataract evidence gap maps (EGMs).
Overall, 30 reviews provided strong evidence in response to the research question, 14 reviews showed weak or no evidence and in 14 reviews the results were inconclusive. Out of 20 reviews examining clinical care, 12 concluded on the presence of strong evidence, 6 showed no or weak evidence and in 2 reviews the results were mixed. In the treatment-related reviews, the strength of evidence was equally split between strong, weak and inconclusive categories.
Using the SURE checklist criteria, 16 reviews were regarded as high quality, 13 were medium quality and 29 were graded as low quality. The majority of high-quality reviews (38.9%) were treatment-related, followed by quality of care, where a third of the reviews were regarded as high quality. Among 30 reviews, which showed the presence of strong evidence, 7 (23%) were high quality and 16 (53.3%) were low quality. Among 14 reviews, which showed weak or no evidence in response to the research question, 6 (42.8%) were high quality and 5 (35.7%) were low quality. Among reviews examining clinical care, only 2 out of 12 reviews showing strong evidence were regarded as high quality; among six reviews showing weak or no evidence, four were high quality. Reviews were attributed low confidence mainly for not reporting the methods used to search the literature or to include studies or extract data, or for not conducting comprehensive searches of the literature.
Clear gaps on evidence were identified on the broader aspects of cataract-related health systems. No reviews were identified for cataract-related screening, and very little evidence is available on the impact of cataract interventions on economic productivity or quality of life. Only three reviews examined cost-effectiveness of cataract-related interventions; all showed strong evidence but the quality of the reviews were mixed (figure 4).
Cataract evidence gap map (EGM).
A total of 47 out of the 52 studies were appraised using both the SURE and SIGN tools; two reviews written in German and three in Chinese were excluded from the SIGN assessment. To determine the extent of reproducibility and the level of agreement between the two tools, we conducted a Kappa statistics test (κ), which resulted in κ=0.79 indicating excellent agreement between the two checklists. Comparison between tools showed that approximately 93% of the reviews that were attributed high confidence in conclusions using SURE were also rated as high quality using SIGN. Sixty-eight per cent of the reviews that were attributed low confidence in conclusions using the SURE checklist were rejected for further evaluation based on the SIGN checklist appraisal (table 1, see online supplementary table S1).
Results overview of the Supporting the Use for Research Evidence (SURE) checklist validation exercise against the Scottish Intercollegiate Guidelines Network (SIGN) checklist
Supplementary tables
Discussion
This paper describes results and experiences of developing the first gap map in eye health, which displays systematic review evidence available on cataract. The majority of the reviews available synthesise evidence on different types of treatment and aspects of clinical care. A smaller number of reviews examined epidemiology, risk factors, access to services and cost-effectiveness of cataract. Only two reviews synthesised evidence on impact of cataract interventions and none focused specifically on cataract-related health systems.3 The conclusions on the strength of evidence reported in the reviews and the rating of the quality of the reviews themselves show mixed results.
With regard to the process of the EGM development, our insights and observations are unique for two reasons. First, this is the first time the EGM approach was applied in the area of eye health. Second, the development of the map was led by an international NGO with contributions from academic institutions, which provided opportunities for integration of both academic and practitioners' perspective in the creation of the map.
In line with the experiences of others, we found EGMs to be a useful tool for presentation of evidence applicable to a variety of audiences, including policy makers and programme managers. Although gap maps do not draw any conclusions or recommendations for policy and practice per se, they allow users to explore the availability and consistency of evidence and to develop their own judgments and implications for practice.10
The EGMs should be analysed with caution to avoid misinterpretation. It is important to take into account that the cataract EGM presented here displays evidence available through systematic and literature reviews only. Neither primary studies nor impact evaluations were included in this EGM. The gaps in evidence presented in our EGM do not necessarily imply the lack of studies but the lack of systematic or literature reviews. It is also important to distinguish between the areas where no evidence exists and therefore more research is needed, and those where primary studies are available but need to be synthesised in a systematic review. As the EGM search strategy is usually comprehensive and includes primary studies, integration of such additional sources of evidence into the same or a separate map should be possible. However, as the process of appraising studies can be very time consuming, this may be more appropriate for the thematic areas where few systematic or literature reviews are available.
When considering evidence presented on the maps, users should take into account the methodological quality of the reviews included.13 Review authors of studies included in the cataract EGM raised concerns about the validity of the SURE checklist due to the lack of information on the validation of this tool. The validation process involved appraising studies included in the EGM using SIGN to assess the consistency and reliability of the SURE checklist tool. Based on the kappa statistics test, results suggested that the SURE checklist is a valid tool to appraise the quality of the reviews for the purpose of EGMs. Implying rigorous development of the tool, items included are based on empirical evidence and acceptable reliability.14
It is important to understand that the appraisal of the quality of the reviews for EGMs is based on the review methods described in the publication and as per the criteria described in the SURE checklist, which are relatively strict. Nevertheless, it is important to note that the medium or low rating of the review does not mean that the study conclusions are not valid. The rating means that we are less confident that they are valid.
In conclusion, EGMs are a useful tool, which provides a one-stop shop of evidence for global policy, advocacy, practice and research. It is a tool through which relationships with various stakeholders involved in international development can be improved and strengthened.
Acknowledgments
We show our appreciation to Catherine Pellegrino, Heiko Philippin, Tingting Chen and Dr Xiao who contributed in the analysis of reviews written in languages other than English. We also thank Kolawole Ogundimu and Clare Gilbert for contributing towards the development of the cataract platform.
Footnotes
Contributors BV contributed towards the conception and design; drafted the manuscript and approved the final version. EJ and ES contributed to the content of the paper, reviewed it critically for substantial intellectual content and approved the final version. IG and CB provided their input towards the paper.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.