Introduction

Cataract surgery is the most frequently performed surgical procedure in the NHS, with 332 625 operations having been reported for 2009–2010 in England, a crude surgical rate of around 6.65/1000 population.1 Ophthalmology departments generally and cataract units in particular are well advanced in the adoption of electronic working practices with ∼40% of English NHS Trusts using electronic patient record systems as part of routine cataract care. The NHS Information Standards Board has recently approved the Cataract National Data set (CND) as an ‘Inherited Data Standard’, which should underpin and facilitate further uptake of electronic working in cataract surgery.2 Routine standardised electronic data collection in the context of a high-volume surgical procedure and a widely accepted intra-operative surgical benchmark quality indicator makes cataract surgery an ideal candidate for the development and piloting of a methodology for case-mix adjustment of outcomes.

Based on analysis of 55 567 cataract operations, our group has reported updates and benchmarks on surgical practice across a range of issues including patient demographics, preoperative visual acuity, anaesthesia, anticoagulants, surgical complications, risk indicators for posterior capsule rupture, ocular biometric properties, visual acuity outcomes and variations in unadjusted posterior capsule rupture rates for individual surgeons.3, 4, 5, 6, 7, 8 Of particular relevance to the present report is our previous finding that based on routine preoperatively collected data, the predicted probability of a complication during surgery may vary by 100 fold from 0.75% for the most straightforward lowest-risk cases to over 75% for the highest-risk most complex cases.5 Furthermore the grade of surgeon has an important influence on the likelihood of a complication arising.9

Transparency of medical and surgical outcomes has become a matter of public concern, with the medical director of the NHS having indicated a clear expectation that in the future the clinical outcomes of individual surgeons will be placed in the public domain.10, 11 The issue is therefore no longer whether this will or will not occur, it is now a question of which form it should take. Risk adjustment of outcomes for case complexity has been in use in cardiac surgery for a number of years.12

The aim of this report is to outline a methodology for case-mix adjustment of surgical outcomes for individual cataract surgeons using electronically collected preoperative multi-centre data conforming to the cataract national data set (CND).

Method

Risk indicators for the benchmark intra-operative surgical complication, posterior capsule rupture or vitreous loss or both (abbreviated in this paper as PCR) have previously been published for this population of cataract operations.5 For the purposes of this report, we repeated the previous multivariate logistical regression analysis with inclusion of information on paired eyes to acknowledge within patient inter-eye correlation. Using risk profiles based on patient and eye indicators applicable to individual operations, a risk score was calculated in the form of a composite odds ratio for each operation. From the statistical model a predicted probability of a complication was derived as described in our earlier report.5 For each surgeon contributing >25 procedures an average case complexity profile was calculated as the mean of the predicted probabilities for PCR across all of that surgeon's cases and this information is presented graphically for all surgeons. As an initial step, based on both this summary measure of the surgeon's case complexity and the grade of the surgeon, the expected complication rate for the cases undertaken was compared directly with each surgeon's actual observed complication rate, the results being presented graphically as the difference between observed and expected rates. Finally, to allow fairer comparison against an average or benchmark rate, complication rates for each surgeon have been adjusted for case complexity.13 To make such adjustment, with placement of confidence limits around the results, the statistical methodology we have used involves construction of funnel plots with the observed/expected rate for each surgeon on the vertical axis against the expected number of complications (PCR events) on the horizontal axis.14 As output from this method may not be intuitively accessible to surgeons unfamiliar with this statistical approach, we have placed a secondary horizontal axis above each plot as an aid to interpretation of the funnel plots. The secondary horizontal axis shows the approximate surgical volume based on the average PCR rate for the relevant surgical group, which is more similar to our previously published unadjusted results.4 In this way separate plots have been presented for: (1) all contributing surgeons (this analysis includes adjustment for grade of surgeon); (2) independent surgeons (consultants, associate specialists, staff grade surgeons); (3) senior trainee surgeons (fellows, specialist registrars) and; (4) junior trainee surgeons (designated as senior house officers at the time of data collection, now first or second year specialty registrars (ST1 and ST2).

A local ethics committee confirmed that ethics approval was not required, as this work was an audit and no patients, hospitals or healthcare workers were identifiable.

Results

There were 283 surgeons with data on >25 cataract operations, together providing data for analysis on a total of 54 319 operations. Inclusion of information on paired eyes to acknowledge any within patient inter-eye correlations improved statistical rigour, with results remaining essentially unchanged from our previously published logistic regression model.5

Case complexity description

Based on the model the average case complexity (predicted probability of PCR from patient factors) for each of the 283 individual surgeons appears in Figure 1, the average predicted probability of a complication varied between 2.62 and 1.17%.

Figure 1
figure 1

Average case complexity profiles for each surgeon's case-mix calculated as if all operations had been carried out by a consultant for each of the 283 eligible surgeons contributing >25 cases.

Full size image

Following inclusion of the risk associated with the surgeon's grade, the arithmetic difference between observed PCR rate and expected rate is plotted against surgical volume in Figure 2 for all 283 surgeons. It is of note that towards the left where samples are small there is a wide vertical spread, which reflects statistical sampling error for small samples.

Figure 2
figure 2

Difference between observed PCR rate and expected PCR rate for 283 surgeons contributing N>25 cases, total N=54 319 operations. Positive % values indicate a rate higher than that expected (poorer than expected performance), negative values indicate better than expected performance. The wide spread to the left demonstrates uncertainty where number of cases is small.

Full size image

Case complexity adjustment and confidence limits

Results are presented as a series of case-mix adjusted funnel plots with each point on the plots representing an individual surgeon's case-mix adjusted results set in the context of their peers and with 95 and 99.8% confidence limits. Figure 3 shows all 283 surgeons with results corrected for both case complexity and grade of surgeon, the overall PCR rate being 1.92%. Figures 4, 5 and 6 show the three grades of surgeon separately, 108 independent surgeons, 140 senior trainees, 35 junior trainees, with results adjusted for case complexity alone and the relevant group PCR average rates taken into account (1.41, 2.48 and 5.1% respectively). On the basis of statistical sampling error it would be expected that 95% of surgeons’ results would lie inside the inner pair of confidence limits with 2.5% above and 2.5% below. Similarly for the outer pair it would be expected that 0.1% (∼1/1000) would lie above and 0.1% below these lines.

Figure 3
figure 3

Adjusted funnel plot for all surgeons: ratio of observed to expected number of operations complicated by PCR vs expected number of complications for each of 283 surgeons. Risk adjustment accounts for patient characteristics and grade of surgeon. A secondary ‘x-axis’ above the plot indicates the approximate volume of cases based on the overall PCR rate=1.92% (total N=54 319 operations).

Full size image
Figure 4
figure 4

Adjusted funnel plot for independent surgeons: ratio of observed to expected number of operations complicated by PCR vs expected number of complications for 108 independent surgeons. A secondary ‘X-axis’ above the plot indicates the approximate volume of cases based on the overall PCR rate=1.41% (N=36 485 operations).

Full size image
Figure 5
figure 5

Adjusted funnel plot for senior trainee surgeons: ratio of observed to expected number of operations complicated by PCR vs expected number of complications for 140 senior trainee surgeons. A secondary ‘X-axis’ above the plot indicates the approximate volume of cases based on the overall PCR rate=2.48% (N=15 277 operations).

Full size image
Figure 6
figure 6

Adjusted funnel plot for junior trainee (SHO) surgeons: ratio of observed to expected number of operations complicated by PCR vs expected number of complications for 35 junior trainee surgeons. A secondary ‘X-axis’ above the plot indicates the approximate volume of cases based on the overall PCR rate=5.1% (N=2557 operations).

Full size image

Discussion

This work pilots a methodology for case-mix adjustment of cataract surgical outcome for the benchmark intra-operative complication of PCR. The methodology is applicable to high-volume medical work with discrete well-defined outcomes. The methodology requires an initial analysis of a large (statistically powerful) set of data in order to identify the risk indicators for outcomes of interest. The rarer the outcome the larger the database required. In terms of this analysis for cataract surgery, our risk model requires validation on a fresh set of data, which we intend to do as a ‘next step’, and looking ahead to the future, all similar health-related risk models should be validated on a regular basis to maintain currency. Our results include descriptions of the overall case-mix risk profiles for each of the 283 surgeons who contributed >25 cases towards the 54 319 cases included in these analyses. On the basis of the patient and eye risk indicators (without adjustment for the grade of operating surgeon), the predicted probabilities for PCR ranged from 2.62% for the surgeon with the most complex overall case mix to 1.17% for the surgeon with the least.

We have chosen >25 cases as the case volume cut off on the basis that an expected number of complications of <0.5 becomes intuitively meaningless as actual complications arise in integer values, and furthermore confidence limits cannot be computed for expected numbers <1 (on the graphical presentations the confidence limits stop at expected number of complications=1). The wide scatter associated with low numbers of cases is illustrated in all the funnel plots presented here. In the absence of a statistically robust method for assessment of complication rates when the number of cases is small, a common sense approach is required. This may for example take the form of a pragmatically chosen rate for low volumes of surgery by an individual surgeon. In the context of appraisal for independent surgeons such an approach would facilitate discussion where deskilling because of low surgical volume was potentially a problem, and would also provide a ‘safety net’ for detection of a possible poor performer undertaking occasional surgery. The rate chosen for low-volume surgery would of necessity be somewhat arbitrary and would need to acknowledge the surgical circumstances so that for established surgeons a slightly different approach would be needed compared with trainees. A candidate for the chosen threshold could be the rate of the upper confidence interval where the line ends towards the left side of the plot (ie, where the expected number of complications=1; Obs/Exp=9.2; Observed Rate=1.41 × 9.2=13%).

This methodology allows the work of individual surgeons to be seen in the context of their peers, both for trainees and established independent surgeons. Confidence limits on the case-mix adjusted funnel plots provide the fairest available method for identification of surgical outliers. In the setting of a surgical procedure where the predicted probability of a complication is estimated to vary by as much as 100 fold depending on patient characteristics, adjustment for case complexity of the case mix of individual surgeons is essential for credible comparisons between surgeons. It should be remembered that with all such analyses statistical sampling error will play a role and it is therefore important to bear this in mind when interpreting such data. In terms of identification of poor performance, a surgeon whose results lie above the upper confidence limits has a probability of being in these extreme upper positions of 2.5 and 0.1%, respectively. Although potentially alarming, such results for individual surgeons must be expected from time to time to arise purely as a result of chance. Interpretation of such results is beyond the scope of this paper but it may be reasonable to suggest that criteria for an amber alert might be outside the upper 95% limit (probability 2.5% or 1 in 40) and for a red alert outside the upper 99.8% limit (probability 0.1% or 1 in 1000), with such alerts based on well defined case-mix adjusted outcomes and analysed separately according to grade of surgeon. On the other hand the opposite tails (the other 2.5 and 0.1%) would be of interest in terms of identification of outstandingly good performers from whom lessons in good practice may be gained.

Increasing expectations on the part of the public have stimulated discussion around the need for transparency in terms of quality of services, including quality of surgical outcomes. In the NHS cardiac surgeons have led the way11, 12, 13 in the wake of well-publicised failures of surgical care. The advent of electronic patient records for routine collection of detailed standardised data items brings fresh opportunities for in depth analysis of specific outcomes including risk indicators for adverse outcomes. These developments open the door to a range of quality improvement opportunities, for example early identification of higher-risk cases to ensure that more risky operations are performed by the most highly skilled surgeons. In the context of revalidation, expected to commence in 2012, the ability to undertake continuous electronic audit will significantly reduce the burden on surgeons to provide evidence of ongoing competence. Case complexity is believed to be a significant determinant of outcome in many medical fields but until recently detailed high-quality information with sufficient statistical power to quantify these risk indicators has been unavailable. Our previous and current work illustrate both the importance and the feasibility of risk adjustment when attempting comparisons between surgeons. Appropriate case-mix adjustment of outcomes should alleviate potential concerns on the part of those highly skilled surgeons who take on higher risk complex cases. Without risk adjustment there is the possibility that surgeons may become risk aversive when offering surgery, with the unwanted consequence that patients, who may otherwise benefit from surgery, could be denied access because of a surgeon's concerns about being adversely judged should a complication arise.

Conclusions

In this paper, we present a methodology for risk adjustment of PCR rates in cataract surgery applicable to individual surgeons. Regular validation of the underpinning risk model should be undertaken to maintain relevance. The methodology has applicability to other fields of high-volume medical and surgical work with discrete outcomes. Properly audited and risk-adjusted surgeon's results will have value in the contexts of fairly judged revalidation and quality assurance for the public.