Background Trachoma remains a significant cause of blindness in many parts of the world. The major route to blindness involves upper lid entropion leading to trachomatous trichiasis (TT), which promotes progressive corneal opacification. The provision of surgery to correct TT in the populations most severely affected is a major challenge for the global effort to eliminate trachoma blindness by the year 2020. Most attention has focused on increasing the quantity of TT surgery performed, and large numbers of non-doctor operators have been trained to this end. Surgical audit by those performing TT surgery is not a routine part of any national trachoma control programme, and no effective mechanism exists for identifying surgeons experiencing poor outcomes. The authors propose a methodology for surgical audit at the level of the individual surgeon based on Lot Quality Assurance.
Methods A set number of patients operated on previously for upper eyelid TT are examined to detect the recurrence of TT. The number of recurrent cases found will lead to categorisation of the TT surgeon to either ‘high recurrence’ or ‘low recurrence’ with reasonable confidence. The threshold of unacceptability can be set by individual programmes according to previous local studies of recurrence rates or those from similar settings.
Conclusions Identification of surgeons delivering unacceptably high levels of recurrent TT will guide managers on the need for remedial intervention such as retraining.
- trichiasis surgery
- Eye lids
- public health
- treatment surgery
Statistics from Altmetric.com
Trachoma is the leading infectious cause of blindness worldwide. Recurrent infection by Chlamydia trachomatis leads to progressive tarsal conjunctival scarring, entropion and trichiasis. Ultimately, corneal blindness develops as a result of the trauma from trichiasis and secondary bacterial infection. Trachoma control programmes (TCPs) in endemic countries seek to prevent blinding trachoma through implementation of the SAFE Strategy: Surgery for trichiasis, Antibiotic distribution to reduce the reservoir of infection, Facial cleanliness and Environmental improvements to reduce transmission.1
Current estimates suggest there are eight million people with trachomatous trichiasis (TT) in need of surgery.2 In response to this large backlog of unoperated TT the emphasis has understandably been on quantity rather than quality of surgery performed. Most trachoma endemic countries do not have enough ophthalmologists to deliver TT surgery and therefore have trained many non-doctor trichiasis surgeons. A randomised trial comparing ophthalmologists with integrated eye care workers found comparable results.3
The usual outcome measure for TT surgery is the proportion of operated eyes with recurrent TT, reportedly ranging from 10% at 1 year to 60% at 3 years.4–10 Two categories are distinguished: early (0–3 months) recurrence due to surgical factors and late recurrence due to progressive cicatricial disease. Several factors may affect the risk of early recurrence: quality of surgery, specific technique, suture type and preoperative disease severity. One study, conducted under ‘operational’ conditions, reported significant intersurgeon recurrence rate variability (0 to 80%).6 However, despite this, little attention has been paid by TCPs to the need for surgical quality assurance. To our knowledge, no formal methodologies have been published to guide this quality-assurance process.
In 2003–2004, evaluations were conducted of the TCPs of eight countries. The most frequently proposed method of surgical quality assurance found was supervisory visits by experienced surgeons to observe TT surgery in progress. However, no documentation of any such supervisory process was found in any setting.11 Such a method of quality assurance may have some validity, particularly if the use of a checklist is employed,12 but is subject to numerous logistical constraints and does not evaluate actual surgical outcomes.
Here we propose a simple quantitative method of surgical audit which can be used by individual surgeons or programme managers to determine with reasonable certainty whether a recurrence rate higher than that deemed acceptable is encountered.
We have used the Lot Quality Assurance Sampling (LQAS) methodology13 to develop an audit tool for assessing the outcome of upper eyelid TT surgery in terms of postoperative recurrence of TT. This method does not estimate the actual recurrence rate experienced, but merely assigns the surgeon or group of surgeons performing the surgery to one of two groups: ‘high recurrence’ or ‘low recurrence.’ This may provide programme managers with useful information to trigger remedial measures such as additional training and supervision where ‘high recurrence’ is identified.
Definitions of recurrence
Standardisation of the definition of recurrence is essential for the audit process. The common definition of recurrent upper eyelid TT after surgery is: one or more upper lid eyelashes touching the globe, evidence of epilation or a history of repeated TT surgery.6 7 10 Some TCPs have a policy of operating only on those with more extensive trichiasis (‘minor’ vs ‘major’ TT).6 Such programmes could use an alternative definition of recurrence: ‘TT requiring surgery,’ similar to that used to drive the initial management decision regarding surgical intervention.
Trichiasis surgery outcome standards
A wide range of TT recurrence rates have been reported.4–8 10 14 Various local factors such as variations in the relative severity of preoperative TT, ongoing exposure to C trachomatis infection, quality of surgical training, volume of surgery being performed, surgical technique and suture material may influence the long-term results.4 9 10 It is therefore difficult to propose a benchmark level for a universally ‘acceptable’ recurrence rate, or suggest categorised benchmarks based on a combination of these factors. Each individual programme would need to determine its own benchmark for maximum acceptable recurrence rate, preferably based on previous studies of recurrence rates or those from similar settings.
Recurrence rates increase with time; hence the interval between surgery and follow-up heavily influences the recurrence rate observed. The lowest published recurrence rates are around 10% at 1 year.7 15 However, most series under operational rather than trial conditions are between 20% and 40%. Taking a pragmatic stand-point, we would suggest that everyone performing surgery for TT should be aiming to achieve a recurrence rate below 20% in the first year postoperatively. This figure does not represent the desired average result for surgeons within a programme, but rather a recurrence rate above which it would be reasonable for a surgeon to be required to undergo a retraining and certification process.
LQAS employs a relatively simple statistical method, which has been reported previously.13 16 17 Software (SampleLQ v1.10) using published algorithms is freely available online (http://www.brixtonhealth.com). In brief, the proportions of the variable under consideration in a given population (recurrent TT) are chosen that are deemed definitely acceptable and unacceptable.
If the benchmark TT recurrence rate is set at 20%, as suggested above, these levels could be set at 10% (definitely acceptable) and 30% (definitely unacceptable). A sample size is then chosen that is small enough to be practicable, but large enough to give an acceptable degree of confidence for categorising a population as experiencing ‘acceptable’ or ‘unacceptable’ recurrence. Utilising SampleLQ v1.10, we determined that a sample (n) of 40 eyes would give around 95% confidence in correctly categorising a sample if the threshold number of cases of recurrence (d) was set at 7. Thus, a sample of up to 40 operated eyes would need to be examined for recurrent TT until either: (1) eight cases with recurrence have been found or (2) all 40 have been examined with seven or fewer recurrent TT cases found. Where 7 (d) or fewer have recurrent TT, the prevalence of recurrence in the sampled population can be classified as 10% or less with a probability of 96%. When an eighth case of recurrent TT is found, the prevalence of recurrence in the sampled population is classified as 30% or more with a probability of 94%.
The risk of wrongly classifying a poorly performing surgeon who is actually experiencing a high recurrence rate (30% or more) as having a low recurrence rate (10% or less) is 4%. This is the ‘consumer probability of error.’ The risk of wrongly classifying a surgeon with a low recurrence rate (10% or less) as having a high recurrence rate is 6% (table 1). This is the ‘provider probability of error.’
In the bottom two rows of table 1 (Benchmark 20% and 30%), the gap between the categorisation prevalence classes is set at 20%. As can be seen for the example with a benchmark of 15%, narrowing this gap increases N and therefore increases the logistical problems of conducting the audit. Widening the categorisation gap will reduce N but leads to an increase in the expected errors. The exact change in accuracy can only be calculated by simulation models or field trials.
The time between surgery and follow-up is important.18 Therefore, efforts should be made to ensure that the mean follow-up interval is approximately 1 year after surgery, and that cases are not included that are outside an acceptable range, such as 10–14 months. In low surgical volume practices, where insufficient cases fall into the 12-month follow-up window (10 to 14 months), the audit could be performed in more than one block, for example, leaving a 4-month gap between each round of recruitment until sufficient cases have been examined.
Patient record keeping
An audit process in any setting is dependent on sufficiently detailed and accurate surgical records to allow tracing of patients. Where adequate record keeping has not been maintained, commencement of this practice would be of first importance. A record should include: name, contact information, age, preoperative disease severity, date of surgery, surgeon name, type of surgery and complications.
Selection of patients to audit
Ideally, the method for selecting cases for audit should minimise potential for bias, while being as tightly clustered around 1 year postsurgery as possible. As a minimum standard, 40 consecutive cases, which tightly cluster around 1 year postsurgery, should be selected for audit. Cases lost to follow-up should be replaced in the sample by a predetermined method, such as the construction of a reserve list of a size sufficient to cover the anticipated number of cases lost to follow-up.
Alternatively, to reduce potential bias further, a list of all cases eligible for inclusion could be constructed, and names selected at random using random number tables or drawing lots. The measures that should be taken to trace cases should be predetermined to minimise bias from excess loss of cases who may be at risk of higher recurrence rates, for instance due to living in more remote areas or having persistent symptoms and being reluctant therefore to reattend for examination. Cases from the original selection that are lost to follow-up should be replaced by the same method of random selection.
Removing the responsibility for case selection and follow-up examination from the surgeon being audited is highly desirable, to ensure greater accuracy in the diagnosis of recurrent TT and to reduce selection bias. This may require training of specific individuals to act as audit workers within a programme.
Transportation and other logistical considerations may favour the audit of the most accessible cases. Unfortunately, this may introduce significant bias towards finding better surgical outcomes. People with easier access to surgical services at health centres may tend to have surgery when the disease is milder (which has a better outcome) and their communities may have benefited from more interventions against trachoma in general.
Many healthcare workers who perform TT surgery are involved in other programmes such as health education, outreach activities, vaccination programmes or antibiotic distribution for trachoma. Data collection for this TT audit process could be combined with activities whereby communities are visited from which cases of TT were drawn, and would permit assurance that all cases were operated on by the same surgeon.
Community-based trichiasis surgery programmes have been established in many countries to provide treatment to millions of people with potentially blinding trichiasis, often in remote rural settings. This has generally involved the training and equipping of healthcare professionals who usually have little or no previous surgical experience. Inevitably, to date, the emphasis has been on delivering high-volume surgery to address the backlog. In recent years, results from operational settings indicate that the outcome of surgery, in terms of recurrent trichiasis, are not as good as anticipated. Therefore, the important issue of surgical quality needs to be seriously addressed by TCPs. Efforts are being made to improve the quality of the surgical technique and the training programmes. A certification process, conducted at the completion of surgical training, has been taken up by some countries.19 However, after this point, there are no formal guidelines on estimating the level of recurrence.
Published data from operational settings indicate that the outcome of TT surgery can vary widely between surgeons.6 The impact of poor-quality surgery may be profound. Repeat TT surgery is technically more difficult. Recurrence probably leads to an increased risk of blindness, increasing reluctance of communities to accept surgery and increased cost to the patient and the health service. Audit is standard practice in modern healthcare systems worldwide. It supports the ongoing improvement and maintenance of standards in services. Therefore, it seems appropriate for TCPs to develop mechanisms for the ongoing audit of results. However, this aspiration comes with many challenges, not least the logistical difficulty of following up patients in inaccessible rural communities. Any proposed methodology should be simple and, where possible, minimise the sample size required.
Methodologies that allow categorisation without absolute definition of rates have previously been used to good effect in blindness prevention. Rapid Epidemiological Mapping of Onchocerciasis surveys in communities suspected of being endemic for Onchocerciasis utilised the LQAS method to determine whether mass distribution of ivermectin is indicated.20 A method based on LQAS has also been employed to prioritise communities in trachoma endemic areas for implementation of trachoma control interventions.13 17 Here we use a similar methodology to stratify the outcome of an individual surgeon's surgery into low- and high-recurrence groups. We believe that this is a practical solution permitting targeted interventions to improve quality. At the same time, it minimises the amount of patients who would need to be followed up.
Probably the biggest challenge in introducing any surgical audit tool would be in empowering the TT surgeons themselves to own the process as a mechanism to facilitate either affirmation of good results or the opportunity to improve. It is important to avoid it being viewed as a tool to identify those who are ‘failing.’ This is very important, as most TCPs are working with limited human resources, and maintaining staff motivation is essential. However, TCPs have a duty of care to provide surgical services with acceptable recurrence rates. For this, it is necessary to monitor results and provide additional training, support and supervision where results are not as good. This proposed methodology would allow an audit to be undertaken in a relatively easy but statistically robust fashion, at the level of the individual surgeon. The methodology might also be applied at the level of a geographic region or for the TT surgical programme as a whole, if more thorough evaluation of postoperative TT recurrence rates were not possible, although it would be more difficult to target remedial action to improve the situation if a higher recurrence rate were identified.
Funding JCB is supported by CBM. MJB is supported by a grant from The Wellcome Trust (080741/Z/06/Z).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.