Article Text
PDF
Abstract
Aims To examine the effect of rounding of visual acuity data on estimates of surgical performance.
Method In this observational study, the distribution of 1-year Snellen visual acuity data from 606 endothelial keratoplasties from the NHSBT UK transplant database was analysed. A curve was fitted to the data and used to estimate the frequencies of partly read Snellen lines. The estimates were used to create a virtual database of grafts with Snellen acuities that included individual letter scores. The virtual database was then sampled to produce datasets for 10 virtual surgeons. Various acuity analysis methods were then applied to simulate possible effects of surgeons rounding their data.
Results Rounding of acuity data was found to have a notable effect on estimates of surgeons’ success rates. When a criterion of 6/12 or better was applied, the success rates ranged from 62% to 80% using a conservative method of estimation but ranged from 80% to 94% using a less conservative method.
Conclusions Rounding of visual acuity scores is an important potential bias in outcome data and should be avoided. If rounding is required we recommend that it is carried out conservatively, giving credit only for lines read completely.
- Cornea
- Diagnostic tests/Investigation
- Epidemiology
- Vision
Statistics from Altmetric.com
Introduction
The UK General Medical Council has recently set out the requirements that doctors need to meet in order to maintain a licence to practice through a process termed revalidation. Doctors are required to present evidence to confirm that they meet acceptable standards of practice. For this reason among others, there is increasing interest in the evaluation of surgical outcomes. With fitness to practice requirements and political interest in comparisons between surgeons (‘league tables’), there is increasing pressure on surgeons to submit high-quality data for review. Informed patient choice is a central principle of healthcare delivery in the UK National Health Service. Accurate information is paramount in a patient's decision to undergo surgery following a thorough appraisal of the risks and benefits of the procedure. Surgical outcomes are also of interest to planners and purchasers of healthcare.
Along with surgical complications, visual acuity measurements are widely used for quantification of ophthalmic surgical outcomes for audit purposes. The Snellen chart is the most widely used visual acuity measure in the UK. The limitations of Snellen measurements are well described in the literature.1 In addition, further errors can be induced by ‘rounding’ the data. When recording postoperative outcome, there is no national guidance as to whether acuity values should be ‘rounded up’ or ‘rounded down’ to the closest complete line for data entry.
The aim of this study was to examine the effect of rounding of acuity data on estimates of surgical performance.
Method
Anonymised summary information on postoperative visual acuities was obtained from the NHS Blood & Transplant (NHSBT UK) transplant database.2 Surgeons receiving donor material from the UK Corneal Transplant Service complete a Transplant Record Form at the time of surgery. Following registration of the graft, each surgeon receives follow-up data collection forms on which to record outcomes at 1, 2 and 5 years postoperatively. The Royal College of Ophthalmologists recommends that such data are submitted as part of the revalidation process whereby corneal surgeons retain their licence to practice.3 The follow-up forms ask for ‘day to day’ visual acuity (with habitual correction) and best visual acuity by any method, including pinhole. Surgeons are not required to enter scores for individual letters on partly read lines.
The 1-year ‘day-to-day’ visual acuity results from 606 endothelial keratoplasties were studied (figure 1). These patients received their first endothelial keratoplasty for Fuchs' endothelial dystrophy between April 1999 and December 2010. This time period spans the development and gradual adoption of this procedure in the UK. No exclusion criteria were applied. In order to simulate the effects of rounding of data, the frequencies of partly read lines are required. As these values are unknown, a distribution curve was fitted to the postoperative visual acuity values and then used to interpolate between the whole-line acuity values to estimate the frequencies of partially read lines.
Frequencies of 1-year day-to-day visual acuity outcomes from 606 endothelial keratoplasties from the NHS Blood & Transplant (NHSBT UK) transplant database. CF, count fingers; HM, hand movements; PL, perception of light; NPL, no perception of light.
The NHSBT data are provided in Snellen notation. In order to position the Snellen values on a linear scale for the x-axis, the Snellen values were converted to logarithm of the minimum angle of resolution (logMAR). Since the LogMAR values for each letter in a partially read line needed to be established, these also had to be formulated. This was achieved by dividing the measurement interval (the difference in logMAR) between complete lines by the number of letters in the measurement interval. Table 1 shows the logMAR values assigned to partially read Snellen lines.
LogMAR values assigned to complete and partially read Snellen lines
Plotting the Snellen values against the log of the logMAR values indicated that a simple linear regression model provided an adequate fit to the data. The model distribution was then used to create a virtual database of 606 grafts with acuities that included individual letter scores. The virtual database was then sampled to produce datasets for 10 virtual surgeons. Fifty cases were assigned randomly to each surgeon.
Various acuity rounding criteria were then applied to simulate possible effects of surgeons rounding their data up or down to the closest complete Snellen line.
For present purposes, a ‘day-to-day’ visual acuity of 6/12 Snellen or better after 1 year of follow-up was deemed to be a successful outcome.
A ‘conservative’ rounding method (credit only given for complete lines read) was compared against an ‘optimistic’ method where partially read lines were given credit as being completely read. In terms of logMAR, the conservative acuity criterion for success was better than or equal to 0.3 logMAR. The optimistic criterion was better than 0.48 logMAR. In other words, a score of 6/12 was awarded if any letter on the 6/12 line was correctly identified.
Results
Figure 2 shows the estimated frequencies of partially read Snellen lines (in logMAR notation). Note that the curve fitting was intended to achieve the best fit in the 6/9 to 6/18 range.
Modelled visual acuity distribution. The chart shows the probability (y-axis) of achieving a specified logMAR visual acuity (x-axis). For above purposes: 0 = better than 6/9, 2 = worse than 6/60. LogMAR, logarithm of the minimum angle of resolution.
Table 2 shows the ‘league table’ (comparative success rates) for the 10 virtual surgeons using the conservative acuity criterion.
Success rates using a conservative acuity criterion (logMAR ≤0.3)
The success rates ranged from 62% (surgeon 7) to 80% (surgeon 2). Table 3 shows the success rates when the optimistic acuity criterion was applied.
Success rates using an ‘optimistic’ acuity criterion (logMAR <0.48)
The apparent success rates were notably higher and ranged from 80% to 94%. Table 4 shows the effect on success rates when the five better surgeons (as judged by table 2) retain their conservative method of estimating success but the five poorer surgeons use the more optimistic method.
Success rates with surgeons using varied criteria
The poorer surgeons predominate at the top of the table. Surgeon 7 (the ‘poorest’) rises to third place. Surgeon 2 (the ‘best’) is relegated to fifth place.
Discussion
The present study demonstrates that, while not having attracted attention historically, small changes in reporting criteria can have potentially large effects on estimates of performance.
Although the limits of precision of individual Snellen measurements are approximately ±2 lines, at group level (eg, when comparing the results of two surgeons) a difference of one Snellen line would generate interest. Yet a difference of almost one Snellen line can be produced by rounding the data. Rounding postoperative visual acuity upwards and hence exaggerating the proportion of patients achieving a particular visual standard is counterproductive on numerous levels. Conversely, it is already well recognised that the use of finer measurement scales has advantages in terms of scale precision4 ,5 and implications for statistical power.
If important conclusions are to be made regarding published surgical performance, it is arguable that the same rigorous standards applied to research should be applied to clinical audit. Although best practice is not practicable in every situation, this paper illustrates the problems that can occur when departing from best practice as regards data collection.
This study has several limitations.
The true distribution of postoperative visual acuity values is unknown because the acuity values in NHSBT dataset have already been rounded at entry and may also be subject to other biases not analysed here. Other potential biases include the use or non-use of forced choice testing and the influence of the mode of optical correction. We do not know which individuals entered the data and what protocols were used to test and to round visual acuity. In addition, the data collection by NHSBT is not complete.6 However, it is reasonable to assume that the shape of the acuity distribution curve is a good approximation to the shape of the true curve. For present purposes, the absolute position of the true curve is less important provided that the shape is similar. The effect of rounding is estimated by examining the effect of changing position on the curve. In other words, we assume that the effect being examined is largely independent of the true prevalence values.
Assumptions have been made regarding the number of lines and numbers of letters on each line of the Snellen chart. The manufacturing of Snellen charts has not been fully standardised, with different charts having different numbers of lines and letters per line. Some charts have a 6/7.5 line and some do not.
Although various assumptions had to be made about the corneal transplant data, we were able to examine a prospectively collected high-quality dataset of 103 postoperative acuities of non-phakic eyes from a population sample where acuity was measured under research conditions.7 The proportion of eyes achieving 6/12 when applying an ‘optimistic’ rounding strategy was 92% but was 87% when the more appropriate conservative rounding strategy was applied. This confirms the effect of rounding in a real dataset.
The relative performance of individuals will be determined by their underlying ability but also by sampling. The confidence limits around estimates of performance will of course be greater with smaller sample sizes and the effects of bias from non-reporting are also likely to be significant. In the present study, we made the assumption that all the virtual surgeons were of similar performance (as cases were allocated randomly) and we used the expected variations in samples to illustrate the points made. The present study also demonstrates a substantial effect of random sampling. It is notable that the effect of random sampling together with variations in rounding method could create differences in success rates from 62% to 94% even without taking true differences in surgical performance into account. For a procedure such as corneal transplantation where relatively small numbers are performed, these observations are striking.
It should be noted the present study does not argue the case for or against Snellen versus logMAR for acuity testing. Snellen charts, in the good acuity range, perform similarly in clinical practice to logMAR.8 The key issue highlighted by the present study is how partly read lines are dealt with as regards data entry. Nor is the present study an argument for or against electronic or paper methods of data collection. Robust processes are required for either method.
Recommendations
We recommend the scoring of individual letters in visual acuity assessment. Furthermore, it is self-evident that there should be uniformity in the way acuity is measured and recorded to make data more comparable and reliable.
It is not appropriate to round visual acuity at the points of data collection, data entry and analysis. If rounding is needed for presentational purposes, for example, reporting the percentage achieving a particular visual standard, we recommend the application of a conservative rounding strategy to visual acuity data. In other words, that credit is only given for lines read completely. We see no reason why these principles should not be applied to other vision test results.
Acknowledgments
We are grateful to Mark Jones, Senior Statistician, NHS Blood & Transplant and NHSBT Ocular Tissue Advisory Group who provided the data for analysis.
Footnotes
-
Contributors NAF was responsible for conceptualising the study and overall design. JM was responsible for review of the existing scientific literature. SB was responsible for the mathematical modelling. NAF is the guarantor of the paper.
-
Competing interests None.
-
Ethics approval We were informed by the South Devon Healthcare Research & Development Department that no ethics approval was required for this study.
-
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- At a glance