Statistics from Altmetric.com
Editor,—Referral accuracy is an important measure of primary care effectiveness. It is defined as the proportion of patients referred for a particular condition who are subsequently diagnosed as having that condition (that is, the true positive proportion). Statistically, it estimates the probability that a patient who is referred will have the disease (positive predictive value) and, as with all statistical estimates, the value calculated in any sample will be subject to error, the magnitude of which decreases as the sample size increases. In recent years there has been an increase in the number of publications on the accuracy of referrals by optometrists to ophthalmology clinics. This letter has been prompted by reading some of those concerned with referrals for suspected glaucoma,1-12 but the issue applies generally to estimates of referral accuracy for any disease condition and, indeed, to all measures of screening effectiveness that involve calculation of sample proportions, such as sensitivity, specificity, and so on.
The majority of reports of estimated referral accuracy give no indication of the precision (standard error) of the estimates. Although some reported estimates of accuracy are obtained from relatively large samples of referred patients, others are based on samples of 10 patients or less. For example, Dayan et al 4 and Newman et al 6 each report samples of referred patients (sample sizes 11 and 10 respectively) in which there were no positively diagnosed cases. Using these data to estimate referral accuracy in the population gives 99% confidence intervals of 0 to 38% and 0 to 40% respectively. This means that if one sample of 10 patients shows referral accuracy of zero, then the referral accuracy in 99% of samples drawn from the same population would not necessarily be zero but would be expected to lie between 0 and 40%. Newmanet al,6 in subdividing referrals according to the mode of screening, obtain some accuracy estimates from even smaller samples. They report, for example, that two out of five patients referred on the basis of optic disc + visual field assessment gave a positive diagnosis of glaucoma; referral accuracy of 40%. For this sample the 99% confidence interval ranges from 8 to 83%. Awareness of the lack of precision in small sample estimates of referral accuracy is important for correct clinical interpretation. The comparative effectiveness of different referral strategies or modes of screening should not be judged on the basis of estimates from very small samples. Clinicians should keep in mind the fact that population values for referral accuracy may in some situations be much higher, or indeed lower, than those observed.
It is therefore recommended that authors should routinely report 95% or 99% confidence intervals (CI) for all measures of diagnostic accuracy. When these measures are simple proportions, as is referral accuracy, the general equation for the confidence interval is CI =proportion ± (z× standard error of proportion). In this equation z is the standard normal deviate;z = 1.96 for a two sided 95% CI, orz = 2.58 for a 99% CI. The common simple formula13 for the standard errorsp
of a proportionp is
where n is the sample size; and the confidence interval is then
However a problem with this formula, which is based on a binomial approximation to the normal distribution, is that it can in some circumstances produce confidence limits of less than 0 or greater than 1, when clearly the population proportion must always lie between 0 and 1. This common approximation should therefore be avoided in favour of exact binomial confidence intervals which are available in many statistical software packages. Alternatively, a formula which always gives confidence intervals within the natural limits of 0 and 1, and is also easy to calculate is14: