You are here

Article Text

Article menu

Download PDFPDF

Clinical science
CUSUM: a dynamic tool for monitoring competency in cataract surgery performance
Free
  1. Mohamad Aziz Salowi1,
  2. Yee-Fong Choong2,
  3. Pik-Pin Goh3,
  4. Mariam Ismail3,
  5. Teck-Onn Lim4
  1. 1Department of Ophthalmology, Sarawak General Hospital, Sarawak, Malaysia
  2. 2International Specialist Eye Centre, ISEC, Kuala Lumpur, Malaysia
  3. 3Department of Ophthalmology, Selayang Hospital, Selangor, Malaysia
  4. 4Center of Clinical Research, Kuala Lumpur, Malaysia
  1. Correspondence to Dr Goh Pik Pin, Department of Ophthalmology and Clinical Research Centre, Hospital Selayang, Lebuhraya Selayang-Kepong, Batu Caves, Selangor 69100, Malaysia; gohpp{at}crc.gov.my

Abstract

Aims To apply cumulative sum (CUSUM) in monitoring performance of surgeons in cataract surgery and to evaluate the response of performance to intervention.

Method A CUSUM analysis was applied to 80 phacoemulsification performed by three ophthalmic trainees and one consultant, for the occurrence of posterior capsular rupture and postoperative refracted vision of worse than 6/12 among patients without pre-existing ocular comorbidity. The CUSUM score of each consecutive procedure performed by an individual surgeon was calculated and charted on CUSUM chart. When trainees' CUSUM charts showed an unacceptable level of performance, their supervisors would give feedback and impose closer monitoring of subsequent surgeries.

Results CUSUM charts of the trainees demonstrated an initial upward followed by flattening trend. This reflects learning curves in their process of acquiring competency in phacoemulsification. In contrast, the consultant showed a flat curve indicating an ongoing maintenance of competence.

Conclusion The CUSUM analysis is able to monitor and promptly detect adverse events and trends of unacceptable outcomes in cataract surgery. This objective and dynamic monitoring makes CUSUM a useful audit tool for individual surgeons, but more so for busy consultants who need to supervise trainees.

  • Cataract surgery
  • CUSUM
  • monitoring
  • statistical process control
  • lens and zonules
  • treatment surgery

https://doi.org/10.1136/bjo.2009.163063

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Cataract surgery is the most common procedure performed in ophthalmology and is one of the first procedures, ophthalmic trainees learn and master. Since the outcome of cataract surgery depends on a surgeon's skill, monitoring surgical competency, especially that of trainees, is essential to ensure patient safety and standard of care.

    Conventional methods used in appraising trainees' surgical performances are log book and progress interview. These methods are subjective, based mainly on the number of procedures performed rather than surgical outcomes, and are arbitrary without explicit reference to predetermined standards. Thus, a more robust and objective method is desirable.

    The cumulative sum (CUSUM) analysis is an objective assessment on the outcome of consecutive performances with reference to an agreed standard. It has long been used in manufacturing industries for quality control.1 CUSUM has increasingly been applied in clinical performances in surgical procedures,2–7 anaesthetic procedures,8 9 interventional nephrology10 and diagnostic procedures.11–14 Besides competency monitoring, it has also been applied to assess the effectiveness of wound dressing.15

    To date, the CUSUM analysis has not been applied in ophthalmology. This study presents the usefulness of CUSUM as a novel performance appraisal tool in phacoemulsification with reference to occurrence of posterior capsular rupture (PCR) and poor postoperative refracted vision.

    Material and method

    The CUSUM analysis is a statistical process control tool. The tool scores performance outcome as a quantitative measurement. CUSUM scores of consecutive performance of an individual operator are displayed as a control or line chart, with the x-axis representing the consecutive series of procedures and the y-axis representing the CUSUM score.

    Mathematically, the CUSUM score is determined using the formula:CUSUM Cn=max(0,Cn−1+Xn−k)where C=case; n=number of procedures (in chronological, consecutive order); Xn=outcome measure for the nth procedure. For binary outcome, Xn=0 (success), Xn=1(failure). For continuous measurement, Xn is standardised to have a zero mean and unit standard deviation (SD). k=reference value (a prespecified standard of performance defined in terms of acceptable and unacceptable failure rate).

    At the start, CUSUM C0=0. At the nth procedure, Xn is the outcome measure for the nth procedure. Performance with an acceptable standard has a negative score, and the chart is either flat or slopes downwards. Performance with an unacceptable standard has a positive score, and the CUSUM chart slopes upwards. When consecutive procedures performed by the same operator are at an unacceptable standard, the chart will continue to slope upward until it crosses a line drawn across the chart called the decision interval (h). When this occurs, the CUSUM chart is said to signal an unsatisfactory performance, thus providing early warning for corrective actions to prevent subsequent patients being harmed by adverse outcome resulting from deteriorating or substandard performance.

    After crossing the decision interval, the CUSUM chart is restarted. Restart should theoretically be at 0 or the x-axis. However, in order to obtain a chart which looks like a learning curve, it restarts at a decision interval, which now acts as a new x-axis.

    The decision interval or h is determined by specifying the in-control (IC) and out-of control (OC) average run length (ARL). IC-ARL is the average number of consecutive surgeries required for a CUSUM chart to cross a decision interval or signal during the period when the operator is performing at an acceptable level. This is analogous to Type I (α error) or false-positive error in hypothesis testing. On the other hand, OC-ARL is the average number of procedures performed before the CUSUM chart signals, during the period when an individual is performing at an unacceptable level. It is a measure of sensitivity and is analogous to power (1-Type II (β)) or false-negative error in hypothesis testing.

    A design with short IC-ARL (large type I or α error) is prone to false alarm, while a design with short OC-ARL (high power) will quickly detect poor performance. Ideally, CUSUM monitoring requires long IC- ARL (small type I error) and short OC-ARL (high power) before the chart signals an actual deterioration in performance. Unfortunately, this ideal could not be reached, as a desirably long IC-ARL (small type I error) will lead to unacceptably long OC-ARL (low power). On the other hand, the desired short OC-ARL (high power) will lead to more frequent false alarms (large type I error). Hence, a trade-off is made between them. The amount of acceptable trade-off depends on the nature of the procedure and its adverse event to be monitored, that is, life-threatening complications would require a highly sensitive chart to detect poor performance at the expense of more frequent false alarms.

    Figure 1 shows the standard CUSUM charts for a consultant and a trainee. A consultant is expected to have a flat CUSUM chart indicating ongoing maintenance of competence. On the other hand, a trainee, in the process of acquiring a new skill, is expected to have an initial upward followed by a flattening chart, the so-called learning curve. The degree of the slope is a measure of progress, that is, a greater slope means slower progress. When the chart flattens, it indicates that the trainee has mastered the new skill.

    Figure 1
    Figure 1

    Standard reference cumulative sum (CUSUM) chart showing performances by a trainee and a consultant.

    In this study, the CUSUM analysis was applied to four cataract surgeons, that is, a senior consultant and three trainees in their performance of 20 cases of phacoemulsification with intraocular lens implantation. Trainee 1 was in his final year of postgraduate training, trainee 2 was in the second year of postgraduate training, and trainee 3 had completed his postgraduate training. The adverse outcomes being monitored were the occurrence of PCR during surgery and postoperative refracted vision. Successful cataract surgery was defined as absent of PCR or refracted visual acuity of 6/12 or better by 12 weeks postoperatively. The prespecified standard of performance was derived from aggregated data of 12 798 cataract surgeries reported to the National Cataract Surgery Registry (NCSR).16 The acceptable rate for PCR was less than 5%, and the unacceptable rate was more than 10% of surgery performed. For visual outcome worse than 6/12, the acceptable rate was less than 10%, and the unacceptable rate was more than 20% of patients operated. Patients who had pre-existing ocular comorbidity were excluded from CUSUM analysis. Table 1 summarises the specifications and parameter values used in CUSUM charting for monitoring cataract surgery performance. The trainees' CUSUM charts were monitored closely. When their CUSUM charts crossed the decision interval, they were given feedback, and closer monitoring of subsequent cases was initiated.

    View this table:
    Table 1

    Cumulative sum charting design for monitoring cataract surgery performance

    Results

    The CUSUM charts for PCR for the four surgeons are shown in figure 2. Trainee 1 had PCR in his first case, and his chart crossed the first decision interval. The subsequent two cases were uneventful, and the chart sloped downwards. However, a second PCR occurred in case no 4. As this was the second occurrence of PCR, and it occurred soon after the first, the chart crossed two decision intervals. For the next 16 cases, trainee 1 did not have any PCR, indicating that his performance had achieved an acceptable standard.

    Figure 2
    Figure 2

    Cumulative sum (CUSUM) chart for four surgeons based on the occurrence of posterior capsular rupture.

    Trainee 2, the junior trainee, had three cases of PCR. The first two cases were associated with extended continuous curvilinear capsulorrhexis, and the last case occurred during irrigation and aspiration. The PCR occurred at a gap of a few successful surgeries. His chart showed an upward followed by downward slope and crossed three decision intervals. A review of his cases with PCR did not indicate that they are difficult cases. Trainee 2 had not overcome his learning curve and needed continuous close monitoring.

    The chart for trainee 3 crossed the decision interval at case no 2 and crossed two decision intervals at case no 5. After the fifth case, there was no more PCR, and his chart stayed flat.

    The consultant had no PCR until the last case. His chart was flat initially and crossed one decision interval at case no 20. The PCR occurred during irrigation and aspiration.

    Figure 3 shows the CUSUM charts for postoperative refracted vision. Trainee 1 had two patients who had postoperative vision worse than 6/12, that is, case no 4 who happened to have PCR and high surgically induced astigmatism, and case no 16 who had cystoid macular oedema (CMO). The chart did not cross the decision interval, as the occurrences were far apart. Trainee 2 had three patients with a poor visual outcome. Case no 1 had PCR and CMO, case no 8 had PCR and severe postoperative inflammation, and case no 9 had infective endophthalmitis. Trainee 3 had three patients with poor visual outcomes, that is, case no 7 who had CMO, and case nos 8 and 19, who had surgically induced high astigmatism. The CUSUM charts of trainees 2 and 3 crossed the decision interval, as two of their three failures occurred sequentially. They were both required to review their cases with poor outcome and to find corrective measures to reduce surgically induced astigmatism. The consultant demonstrated a flat chart with one isolated case of poor vision which was due to CMO.

    Figure 3
    Figure 3

    Cumulative sum (CUSUM) chart for four surgeons based on a postoperative refracted visual acuity worse than 6/12.

    Discussion

    Applications of CUSUM in clinical practice, especially in monitoring trainees' performance, have been encouraging. Its usefulness has been documented in endotracheal intubation,8 extradural and spinal anaesthesia,9 endoscopic retrograde cholangiopancreatography,14 18 renal biopsy4 and catheter insertion in peritoneal dialysis.10 This study on phacoemulsification, displaying CUSUM charts of three trainees with a typical learning curve proved that CUSUM has the ability to promptly detect poor performances and is an effective competency monitoring tool in cataract surgery as well.

    Monitoring trainees' surgical performance is a demanding task. The conventional methods are logbooks, progress interviews, peer reviews or direct supervision. Trainees are often assessed based on quantity, that is, the number of surgery procedures performed, and not quality, that is, rates of surgical complication or successful outcome. These methods are subjective, often unreliable, time-consuming and without reference to validated standards. Even if surgical outcomes were monitored by reviewing log book, it is slow in detecting trends of unacceptable performances, and much harm could have been done to patients before the deteriorating performance of a trainee is identified.

    CUSUM is superior to conventional methods, as it is objective and dynamic, tracks performance over time and refers to predetermined outcome standards. The standard of performance can be modified based on updated values derived from benchmarking. Different standards can also be set for surgeons at different levels of training. The unique and important feature of CUSUM is its ability to indicate the trend of deteriorating performance early, prompting preventive and corrective measures.

    The graphic display of CUSUM chart is simple and easy to understand. An upward chart signifies unacceptable performance. When it happens, both the trainees and their supervisors should take action, for example by reviewing video recordings of the surgery and closer monitoring on subsequent surgeries. When the upward trend is continuous despite intervention, this may indicate the need to assess the surgeons' stereo-acuity and bi-dexterity, essential prerequisites in performing microsurgery.

    CUSUM can also be used to set an average number of procedures needed to be done before trainees are allowed to perform independently. Whenever possible, trainees should carry out that set number of procedures on animal or simulated eyes in a wet lab before performing on patients. This application has been studied for surgical trainees for endotracheal intubations, in which 19 attempts were required to overcome the learning curve,8 and for Tenckhoff catheter insertion for peritoneal dialysis, in which 23 procedures needed to be carried out to achieve less than 25% primary failure.10 Failing to achieve the learning curve at the specified number of cases may indicate that the particular trainee needs further supervision before being allowed to proceed to the next level of training.

    CUSUM may not be applicable to all surgical procedures. The efficiency of CUSUM as a monitoring system depends on several criteria. First, the procedures must be frequently performed. Second, the success or failure of the procedure must be clearly defined and is closely related to surgeon's skill. Third, the complications of the procedure must be well defined. Cataract surgery fulfills such criteria, especially for PCR, as its occurrence closely reflects the surgeon's competency, and it is one of the main predictors for poor visual outcomes.19–25 Postoperative visual outcome is an indirect measure of surgeons' competency as patients' factors such as CMO can contribute to poor visual outcome. Thus, when reviewing trainees' CUSUM charts, their feedback on the occurrence of intraoperative complication and factors contributing to poor visual outcome are relevant and should be considered.

    Like other monitoring systems, CUSUM faces the challenge of incomplete data entry on surgeries performed by individual surgeons. This problem can be overcome by automated data mining, either from electronic medical records or from the patient register. Since early 2009, CUSUM competency monitoring has been applied to doctors serving the Ministry of Health of Malaysia whose surgeries are captured in the cataract surgery registry. Individual surgeons can view their own CUSUM charts on the eCUSUM website (https://app.acrm.org.my/eCUSUM). When trainees view their charts online, they can activate an automated email message that will send their charts to their supervisors. By doing so, monitoring becomes more systematic, effortless and in real-time. Monitoring and evaluating the competency of surgeons, especially trainees, are essential and mandatory in ensuring patient safety and standard of care. We are convinced that a CUSUM analysis is useful, especially for busy consultants who need to supervise trainees. Beside the supervisor, individual surgeons can also use it for self appraisal and continuous quality improvement. CUSUM analysis on cataract surgery has now been implemented in the ophthalmology programme in the Ministry of Health Ophthalmology of Malaysia.

    Acknowledgments

    The authors wish to thank the Ministry of Health for the permission to conduct the study and for providing the research grant, N Hisham bin Abdullah, Deputy Director General, Ministry of Health, for his idea and support in the application of CUSUM in the Ministry of Health, P Lum and S Chen for the system development and statistical help.

    References

    1. ↵
      1. Williams SM,
      2. Parry BR,
      3. Schlup MM
      . Quality control: an application of the cusum. BMJ 1992;304:1359–61.
      OpenUrlFREE Full Text
    2. ↵
      1. Jaffer U,
      2. Cameron AE
      . Laparoscopic appendectomy: a junior trainee's learning curve. JSLS 2008;12:288–91.
      OpenUrlPubMed
    3. ↵
      1. Rozenberg P,
      2. Porcher R,
      3. Salomon LJ,
      4. et al
      . Comparison of the learning curves of digital examination and transabdominal sonography for the determination of fetal head position during labor. Ultrasound Obstet Gynecol 2008;31:332–7.
      OpenUrlCrossRefPubMed
    4. ↵
      1. Lim TO,
      2. Soraya A,
      3. Ding LM,
      4. et al
      . Assessing doctors' competence: application of CUSUM technique in monitoring doctors' performance. Int J Qual Health Care 2002;14:251–8.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Van Rij AM,
      2. McDonald JR,
      3. Pettigrew RA,
      4. et al
      . Cusum as an aid to early assessment of the surgical trainee. Br J Surg 1995;82:1500–3.
      OpenUrlPubMedWeb of Science
    6. ↵
      1. Choi DH,
      2. Jeong WK,
      3. Lim SW,
      4. et al
      . Learning curves for laparoscopic sigmoidectomy used to manage curable sigmoid colon cancer: single-institute, three-surgeon experience. Surg Endosc 2009;23:622–8.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Bergman S,
      2. Feldman LS,
      3. Anidjar M,
      4. et al
      . ‘First, do no harm’: monitoring outcomes during the transition from open to laparoscopic live donor nephrectomy in a Canadian centre. Can J Surg 2008;51:103–10.
      OpenUrlPubMed
    8. ↵
      1. Yang A,
      2. Miller JP,
      3. Azaraov K
      . Establishing learning curves for surgical residents using cumulative summation analysis. Curr Surg 2005;62:330–4.
      OpenUrlCrossRefPubMed
    9. ↵
      1. Kestin IG
      . A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995;75:805–9.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Goh BL,
      2. Yudisthra MG,
      3. Lim TO
      . Establishing learning curve for Tenckhoff catheter insertion by interventional nephrologists using CUSUM analysis: how many procedures and in which situation? Semin Dial 2009;22:199–203.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Sibanda N,
      2. Lewsey JD,
      3. van der Meulen JH,
      4. et al
      . Continuous monitoring tools for pediatric surgical outcomes: an example using biliary atresia. J Pediatr Surg 2007;42:1919–25.
      OpenUrlCrossRefPubMed
    12. ↵
      1. Molloy M,
      2. Bower RH,
      3. Hasselgren P,
      4. et al
      . Cholangiography during laparoscopic cholecystectomy. Cumulative summation analysis of an institutional learning curve. J Gastrointest Surg 1999;3:185–8.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Parry BR,
      2. Williams SM
      . Competency and the colonoscopist: a learning curve. Aust N Z J Surg 1991;61:419–22.
      OpenUrlPubMedWeb of Science
    14. ↵
      1. Schlup MM,
      2. Williams SM,
      3. Barbezat GO
      . ERCP: a review of technical competency and workload in a small unit. Gastrointest Endosc 1997;46:48–52.
      OpenUrlCrossRefPubMed
    15. ↵
      1. Chang WR,
      2. McLean IP
      . CUSUM: a tool for early feedback about performance? BMC Med Res Methodol 2006;6:1–5.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Yen SS,
      2. Goh PP,
      3. Mariam I
      . National cataract surgery registry 2002–2003—a report of clinical outcome. Am J Ophthal 2005;139(Suppl 2).
    17. ↵
      1. Green P,
      2. Lawless JF,
      3. Nair V
      1. Hawkins DM,
      2. Olwell DH
      . Cumulative sum charts and charting for quality improvement. In: Green P, Lawless JF, Nair V, eds. CUSUM design. New York: Springer, 1998:31–45.
    18. ↵
      1. Biau DJ,
      2. Williams SM,
      3. Schlup MM,
      4. et al
      . Quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg 2008;95:925–9.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Cruz OA,
      2. Wallace GW,
      3. Gay CA,
      4. et al
      . Visual results and complications of phacoemulsification with intraocular lens implantation performed by ophthalmology residents. Ophthalmology 1992;99:448–52.
      OpenUrlPubMedWeb of Science
    20. ↵
      1. Goh PP,
      2. Rampal SL,
      3. Choong YF
      . National cataract surgery registry 2002–2003—predictors for poor visual outcome. Am J Ophthal 2005;139(Suppl 4).
    21. ↵
      1. Ionides A,
      2. Minassian D,
      3. Tuft S
      . Visual outcome following posterior capsule rupture during cataract surgery. Br J Ophthalmol 2001;85:222–4.
      OpenUrlAbstract/FREE Full Text
    22. ↵
      1. Johansen J,
      2. Theodorsen FP,
      3. Corydon L
      . Visual outcome following complicated extracapsular cataract extraction. J Cataract Refract Surg 1992;18:577–81.
      OpenUrlPubMed
    23. ↵
      1. Lundstrom M,
      2. Stenevi U,
      3. Thorburn W
      . Outcome of cataract surgery considering the preoperative situation: a study of possible predictors of the functional outcome. Br J Ophthalmol 1999;83:1272–6.
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Riley AF,
      2. Malik TY,
      3. Grupcheva CN,
      4. et al
      . The Auckland cataract study: comorbidity, surgical techniques, and clinical outcomes in a public hospital service. Br J Ophthalmol 2002;86:185–90.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Schein OD,
      2. Steinberg ED,
      3. Cassard SD,
      4. et al
      . Predictors of outcome in patients who underwent cataract surgery. Ophthalmology 1995;102:817–23.
      OpenUrlPubMedWeb of Science

    Footnotes

    • Funding This study was supported by Malaysian Ministry of Health Minor Research Grant and is a research project for the first author as partial fulfillment for the degree of Master of Surgery (Ophthalmology), National University of Malaysia.

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Linked Articles