Purpose This study used the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument to evaluate the methodological quality of clinical practice guidelines (CPG) published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO) for the management of cataract in adults.
Study design An evaluation of the AAO, COS and RCO CPGs using a reliable and validated instrument.
Methods Four evaluators independently appraised the three CPGs using the AGREE II Instrument, which covers six domains (Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence). The AGREE II includes an Overall Assessment summarising guideline methodological rigour across all domains, using a 7-point scale where perfect adherence equals a score of 7.
Results Scores ranged from 36% to 75% for the AAO guideline; 45% to 94% for the COS guideline and 23% to 85% for the RCO guideline. Intraclass correlation coefficients for the reliability of mean scores for the AAO, COS, and RCO were 0.78, 0.74 and 0.80; 95% CIs (0.60 to 0.90), (0.45 to 0.88) and (0.53 to 0.91), respectively. The strongest domains were Scope and Purpose (COS, RCO), Clarity of Presentation (COS, RCO) and Editorial Independence (AAO, COS). The weakest were Stakeholder Involvement (AAO), Applicability (AAO, COS) and Editorial Independence (RCO).
Conclusions Cataract surgery practice guidelines can be improved by targeting stakeholder involvement, applicability and editorial independence.
- Lens and zonules
- Treatment Surgery
- Medical Education
Statistics from Altmetric.com
Cataract surgery is the most commonly performed elective surgical procedure and is associated with significant improvement in quality of life.1–3 Therefore, it is imperative that clinical practice guidelines (CPG) for cataract management are developed with transparency and methodological rigour. However, recent studies evaluating the trustworthiness of CPGs have shown that guidelines contain conflicting recommendations, exhibit discrepancies in evidence quality, lack rigour and transparency in their development processes and have potential conflicts of interest (COI) among members of guideline development groups (GDG).4–7 These findings undermine the validity of CPGs, limiting their usefulness as evidence-based resources for clinicians.5 ,8
The Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument9 is a reliable and validated method for evaluating the methodological rigour of CPGs.10–12 With the goal of optimising future CPGs, this study used the AGREE II Instrument to evaluate adult cataract CPGs published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO).
The AGREE II Instrument comprises 23 assessment items, which evaluators rate on a scale of 1 (strongly disagree) to 7 (strongly agree). The 23 items are organised into six quality domains: (1) Scope and Purpose; (2) Stakeholder Involvement; (3) Rigour of Development; (4) Clarity of Presentation; (5) Applicability; (6) Editorial Independence.13 Two final summary assessment items included in the AGREE II rubric enable the appraiser to integrate the various components in formulating a comprehensive evaluation of CPG methodological rigour.
The AAO cataract guideline outlines 11 highlighted recommendations and an additional eight categories of recommendations for the care of adult patients with cataract.2 The COS cataract guideline outlines 65 recommendations organised into six categories.14 The RCO guideline for the management of cataract includes recommendations for both adult and paediatric patient groups, and is organised into 12 categories.3 Four of the authors (CMW, AMW, BKY, DW) independently assessed each guideline focusing on recommendations for cataract management in adult patients, including the AAO cataract guideline supplementary documents, Highlighted Recommendations and Summary Benchmarks. The scores were then averaged and summarised as a scaled percentage score using the following formula recommended by the AGREE II: (Obtained Score−Minimum Possible Score)/(Maximum Possible Score−Minimum Possible Score). An intraclass correlation coefficient (ICC) was used to measure inter-rater agreement for each guideline's scores.
Domain scores are summarised in table 1.
The AAO guideline scored lowest in Domain 2 (Stakeholder Involvement) with a scaled score of 36% and highest in Domain 6 (Editorial Independence) with 75%. The COS guideline scored lowest in Domain 5 (Applicability) with 45% and highest in Domain 4 (Clarity of Presentation) with 94%. The RCO guideline scored lowest in Domain 6 with 23% and highest in Domains 1 (Scope and Purpose) and 4, with 83% and 85%, respectively.
The ICCs for the AAO, COS and RCO were 0.78, 0.74 and 0.80; 95% CIs were (0.60 to 0.90), (0.45 to 0.88) and (0.53 to 0.91), respectively. To determine the strengths and weaknesses of the guidelines summarised in table 2, we assessed the average scores of each domain for uniformity and examined comments for concurrence among at least three of the four evaluators.
This AGREE II analysis highlighted several areas where future CPGs for the management of adult cataract could be improved. All three guidelines fell short in Stakeholder Involvement (Domain 2) by not delineating specific roles of GDG members or the involvement of patient representatives. Diversity among GDG members is integral to formulating less extreme recommendations, preventing the promotion of treatments biased toward a particular specialty, and ensuring the comprehensive inclusion of relevant scientific evidence.15 The AGREE II states that all stakeholders in the care process, including patients, should be involved in recommendation development. It was not possible to determine if patient groups were consulted, with the exception of the RCO guideline, which mentioned public involvement in the guideline development process from the ‘College Lay Advisory Group’. None of the guidelines specified the outcomes gathered from public input. Included within Domain 2 is the requirement for full disclosure of GDG members’ roles and areas of expertise. Although the guidelines addressed GDG members’ areas of expertise, none mentioned their individual roles pertaining to guideline development.
Transparency of the guideline development process and strength of supporting evidence evaluated under Rigour of Development (Domain 3) presented areas for improvement across the board. First, the external review processes were unclear in all guidelines as they lacked descriptions of external review methods, external reviewers, patient input, information gathered from external review and how the outcomes informed the guideline development. In the AAO guideline, the role of the Consumers Union, the only non-medical external reviewing body, was not clearly defined as representing patients or patient advocates.16 If the Consumers Union was intended to represent cataract patients, this objective should be explicitly stated in the guideline. The other guidelines provided no details of the external review process, so it was impossible to verify if public input through the external review played a role in shaping recommendation formulation. Second, the rigour of supporting evidence was variable, including the lack of a clear evidence rating system (AAO) and failure to disclose detailed search terms (COS, RCO). Consistent evidence rating systems as used by the COS and RCO should be implemented in all cataract CPGs because they illuminate the strength of supporting evidence and clarify which statements are recommendations. Similarly, detailed evidence search terms and methods were only available for the AAO guideline. Literature search methods should be explicitly described to enable independent reproduction of developers’ conclusions and to facilitate transparency of guideline development methods.
On Clarity of Presentation (Domain 4), the guidelines employed different methods to ensure utility of recommendations, yet each displayed distinct needs for improvement. The RCO recommendations were flagged throughout the text using boxes, and the COS recommendations were easily identifiable with clear numbers and bold headings. However, neither document provided a concise summary of key recommendations. The AAO guideline provided a supplementary list of 11 major recommendations under ‘Highlighted Findings and Recommendations for Care’, but recommendations in the main text were harder to locate relative to those of the COS and RCO guidelines. Some of the recommendations were primarily summary statements (e.g. ‘cataract surgery is a procedure appropriately utilized in the United States’) rather than actionable recommendations.
Regarding Applicability (Domain 5), the guidelines varied in their consideration for resource implications, auditing criteria and the provision of supplementary documents for guideline implementation. While the AAO guideline included sections on Socioeconomic Considerations, Cost Effectiveness and Cost Considerations, and the RCO guideline included a discussion of problems for the provider, local eye unit and commissioner, overall there was insufficient disclosure of methods by which cost information was sought, and the impact of such considerations on guideline development. The guidelines provided sparse auditing criteria except for the mention that ‘outcome measures should be varied and should match the indications for surgery’ (COS). The intangible influences of differing healthcare systems’ methods of reimbursement are difficult to assess, and it may not be possible to control for the different influences health insurance might have on assessing applicability. Nonetheless, among the CPGs, there was an overall paucity of supplementary algorithms, surgical checklists and other tools to facilitate implementation of recommendations.
All CPGs should maintain Editorial Independence (Domain 6), yet all three guidelines failed to specify methods by which potential competing interests were sought and how the potential COI influenced the formulation of recommendations. The AAO provides a blanket statement online indicating compliance for all its CPGs with the Council of Medical Specialty Societies Code for Interactions with Companies,17 but no specific information on potential COI influence on the development of the cataract CPG. Lack of COI transparency is of particular concern when chair members and a majority of GDG members have financial interests, as was the case for the AAO. Whereas the AAO and COS guidelines mentioned independence of the funding body and provided a list of specific financial disclosures, the RCO guideline provided no mention of the funding body and no comprehensive financial disclosures for GDG members, stating only that the ‘members of the working party have made the Chair aware of their commercial relationships.’
We acknowledge the limitations to this study. First, rating systems involving human evaluators will always limit objectivity, including potential biases associated with nationality. However, the AGREE II system has been demonstrated to be valid and reliable, especially with multiple independent raters, as this study used.10 ,11 All raters completed the AGREE II tutorial which provides clear benchmarks for scoring along the 7-point scale.13 The independent evaluators were highly reliable (based on ICCs of 0.78, 0.74 and 0.80), which is consistent with previous AGREE II studies.9 Second, AGREE II examines the methodological rigour and reporting transparency rather than recommendation validity or evidence quality.9 While rigorous guideline development processes do not necessarily guarantee quality guideline content, suggestions for methodological improvements can be universally applicable and transparency can facilitate evaluation of clinical applicability. Finally, the AGREE II gives equal weight to all six domains, whereas domains may vary in depth and importance. We reconciled this potential issue by using the Overall Assessment and evaluator comments to assess each guideline holistically and identify specific areas for improvement.
In summary, the primary weaknesses of the adult cataract CPGs were in the areas of patient involvement, transparency of the guideline development process, applicability and editorial independence. We have formulated several recommendations for the targeted improvement of adult cataract CPGs, using models from the reviewed CPGs when possible (table 3). This assessment using the AGREE II instrument underscores the importance of regular evaluation of CPGs to optimise patient care.
Presentation Presented in part as a poster at the 2014 Association for Research in Vision and Ophthalmology Annual Meeting, 4–8 May, Orlando, FL.
Contributors The seven authors meet authorship criteria. In detail: CW, AW, BY: study design, analysis and interpretation of data, drafting of the manuscript, final approval given; DW: acquisition of data, analysis and interpretation of data, final approval given; AC: statistical calculations, analysis and interpretation of data, critical review of the manuscript, final approval given; CM: analysis and interpretation of data, critical review of manuscript, final approval given; PBG—study design, analysis and interpretation of data, critical review of the manuscript, final approval given.
Disclaimer The views expressed in this article are those of the authors, and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.
Competing interests PBG has received a USA Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award. The funding organisation had no role in the design or conduct of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.