Background/aims: To assess reliability and validity of the QUOTE-cataract, a questionnaire that measures the quality of care from the perspective of cataract patients.
Methods: The QUOTE-cataract was tested in a multicentre study among 540 cataract patients in three different hospitals. Reliability was represented by internal consistency (Cronbach’s α), and repeatability (intraclass correlation coefficient (ICC)). Validity was evaluated qualitatively and by factor analyses.
Results: A strong internal consistency coefficient (0.89), and high repeatability (ICC = 0.76) demonstrated good reliability. Content validity was assured by involvement of patients in the development of the questionnaire. Factor analysis confirmed an underlying taxonomy of generic and disease specific items.
Conclusion: The QUOTE-cataract has good reliability and provides a valid assessment of quality of care in cataract surgery.
- quality of health care
Statistics from Altmetric.com
In studies assessing the effectiveness of cataract surgery the importance of the patient’s perspective has been recognised. Besides clinical outcomes like visual acuity, assessments are directed at subjective measures of vision related functioning.1,2 However, a valid assessment of the quality of ophthalmic services from the patient’s perspective should consider all components that contribute to the quality of these services—objective outcome criteria (functional tests), subjective outcome criteria (patient satisfaction), and criteria related to the way the services are provided (structure and process of care).3–6 Previous research examining quality of care from the user’s perspective has been dominated by patient satisfaction surveys.7 Doubts have been cast on the validity of such research for two main reasons. Firstly, these studies usually do not involve patients in the development of instruments. Therefore, outcomes tend to reflect the perspective of managers and/or professionals and/or researchers, rather than the distinct view of the patient.8 Secondly, results show little insight into user views because satisfaction ratings are usually highly skewed, while no recognition is given to the fact that some quality aspects of care are more important than others.9,10 This has led reviewers to conclude that research into user views of quality of care has suffered from methodological weakness, and low specificity of results.11–13 We developed the QUOTE-cataract questionnaire (QUality Of care Through the patient’s Eyes) that measures the quality of care from a cataract patient’s perspective, which attempts to overcome these problems. The QUOTE-cataract closely resembles the work that was carried out on other QUOTE-instruments (for example, QUOTE-Rheumatic-Patients and QUOTE-Elderly) as described by Sixma et al.8,14 This instrument produces data on the importance and performance of healthcare services, which are related to the specific needs and expectations of cataract patients. Besides being useful and applicable to clinical practice, an instrument like the QUOTE-cataract should satisfy scientific quality standards. The aim of this study was to assess the reliability and validity of the QUOTE-cataract questionnaire.
PATIENTS AND METHODS
This study consisted of patients who were treated at three settings: University Hospital Maastricht (UHM, n = 166), University Hospital Groningen (UHG, n = 130), and the Rotterdam Eye Hospital (REH, n = 244). The mean age (SD) of the patients was 71.8 (8.0) years (UHM); 73.9 (9.2) years (UHG), and 71.9 (9.3) years (REH), respectively. The male: female ratio was 0.6 for all three settings. At REH more patients were categorised as “medium/higher educated” in comparison with UHM and UHG (Kruskal-Wallis, p<0.01). General health was reported as “good” by most patients, but differed significantly between the three settings with the highest scores at REH (Kruskal-Wallis, p<0.01). Criteria for participation in the study were first or second eye surgery for age related cataract 2–8 months earlier, and absence of ocular comorbidity. All cataract surgeries took place on an outpatient basis using a standardised phacoemulsification technique with implantation of an intraocular lens.
The QUOTE-cataract includes a generic and a disease specific part (see Table 1). To generate these items focus group interviews with cataract patients and concept mapping were performed.15,16 Generic questions (i = 15) are applicable to a range of users of health care (that is, patients) and refer to process and structure related care aspects, whereas disease specific items (i = 16) are tailored to cataract patients. Four disease specific items were not included in the QUOTE-cataract at UHG because they were characteristic for UHM and REH. The items incorporated in the questionnaire measure the relative importance of the different quality aspects (for example, “My ophthalmologist should inform me clearly about what I may and may not do after cataract surgery”) and the performance of the healthcare services on each of the quality aspects (for example, “My ophthalmologist informed me clearly about what I may and may not do after cataract surgery”). Response options of the importance categories are “not important,” “fairly important,” “important,” and “extremely important.” Scores were calculated by linear transformation of standardised values (Z scores) to values between 0 (“not important”) and 10 (“extremely important”).14,17 Response options for perceived performances (1 = “no,” 2 = “not really,” 3 = “on the whole, yes,” and 4 = “yes”) were expressed as proportions of respondents who did not (score 1 or 2) perceive the particular quality aspect as being performed.8 Individual (i) performance (P) and importance scores (I) on different quality of care aspects (j) are used to calculate quality impact indices (Q), applying the formula Qij = Pij × Iij. Theoretically a quality impact score can vary from 0 (best possible quality of care) to 10 (all respondents think that this aspect is extremely important and 100% of the patients report that it needs improvement).
Reliability and validity testing
A postal survey involved 540 patients who had undergone cataract surgery. Cronbach’s α was used to represent the internal consistency of the QUOTE-cataract based on one rating. A “benchmark” of 0.80 was considered as an acceptable value. In accordance with previous reports on the assessment of test-retest stability (repeatability), scoring of the QUOTE-cataract instrument was repeated after 2–3 weeks by 289 cataract patients from the UHG and REH.18 A Bland-Altman plot of agreement and the intraclass correlation coefficient demonstrated variation in test-retest.19,20
Content validity was optimised by involving cataract patients in the development process. Moreover, a question was included to check if patients felt that particular aspects of relevance to them were missing in the questionnaire. Furthermore, factor analysis was performed to examine the taxonomy/structure of the quality of care concept.
Importance scores were used in statistical tests to assess the reliability and validity of the scales. While performance scores will change when situations in healthcare services change, importance scores are less subject to situational changes as they are linked to the attitudes and opinions of patients.8 Cases with more than 10% missing values on the importance scores were excluded from reliability and factor analyses, remaining missing values were replaced by the mean. The Bland-Altman plot indicated stability of test results by demonstrating variation in scores as part of unreliability. One way ANOVA (random effect model) was used to obtain the intraclass correlation coefficient for retest reliability according to the following formula: ICC = 1 − (within subject variance/between subject variance). Factor analysis (principal component) was carried out to represent the internal structure of the questionnaire. The appropriateness of the factor analytic model was tested using the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity. Analyses were performed using spss 10.0 for Windows (SPSS Inc, Chicago, IL, USA) software.
Response to questionnaires
The total number of completed questionnaires returned was 166 (response rate 92.7%), 130 (response rate 65%), and 244 (response rate 74.4%) for UHM, UHG, and REH respectively.
The mean (SD) of the importance scores was similar comparing the three settings: 6.1 (1.6), 6.2 (1.2), and 5.8 (1.2) for UHM, UHG, and REH respectively (ANOVA, p = 0.13). Performance sores ranged from 0.01 to 0.49, from 0.01 to 0.49, and from 0.00 to 0.44 for UHM, UHG, and REH respectively. Importance scores were less skewed in our data set than the perception scores (range of skewness scores −0.58 to 0.30 and 0.43 to 11.3, respectively). Quality impact factors ranged from 0.05 to 3.39 (UHM), from 0.05 to 2.79 (UHG), and from 0.00 to 3.00 (REH), with skewness scores ranging from 0.70 to 1.58.
The mean internal consistency coefficient of the three hospitals was 0.89, 0.84, and 0.85 for the QUOTE-cataract in total, and generic, and disease specific subscales, respectively. The mean (SD) importance scores for first and second assessments were 6.2 (1.2) and 6.2 (1.3) for UHG, and 5.8 (1.2) and 5.9 (1.4) for REH (paired t test, p = 0.89/0.61). The differences in first and second assessments were plotted against the mean QUOTE-cataract score for each patient (see Fig 1), which showed no relation between the measurement error and the true value. The mean of the differences between the pairs was negligible (mean difference −0.04). Furthermore, the limits of agreement were small, an individual measurement difference is expected within a range from −2.34 to 2.34. A strong intraclass correlation coefficient confirmed this consistency (average measure ICC = 0.76).
The appropriateness of the factor analytic model was assessed by KMO (0.92) and Bartlett (χ2= 4131.5, p<0.01), which demonstrated legitimacy of the procedure. Unrotated factor analysis identified two common factors that explained 35.3% of the variance. When analysing generic and specific items separately, both generic and disease specific items clustered into one factor with moderate factor loadings (range 0.45–0.69). Explained variances were 31.6% and 33.7% respectively. The percentage of patients who responded to the question “Were any quality of care aspects missing in this questionnaire?” was 27.1, 26.9, and 22.5 for UHM, UHG, and REH respectively (χ2 test, p= 0.49). However, most answers related to alternative expressions of items already included in the QUOTE-cataract. Real new aspects, formulated by 6,7% of the patients, concerned the anaesthesiology before cataract surgery (n = 3), follow up visits (prescription of glasses and information about secondary cataract or other complications; n = 22), reassurance (n = 4), costs (n = 4), and technical competence (n = 3).
Former research emphasised the relevance of process and structure related quality of care with respect to satisfaction after cataract surgery.1 However, the reliability and validity in measuring the multidimensional concept satisfaction has been criticised frequently based upon the global approach and problems regarding methodological weakness (for example, highly skewed).11–13 The QUOTE-cataract instrument was developed to assess patient’s interests in a more specific and direct way, as recommended by Fitzpatrick.21 We showed that the QUOTE-cataract had low skewness scores in comparison with distributions of 80–90% generally categorised as satisfied.1 The internal consistency of the questionnaire was shown to be as good as the QUOTE-Rheumatic-Patients and QUOTE-Elderly with Cronbach’s α values of 0.92 and 0.93 respectively.8,14 Test-retest data for the QUOTE-cataract confirmed repeatability, there being negligible differences between pairs using the method of Bland and Altman, and a high intraclass correlation coefficient (ICC = 0.76). Factor analysis showed that category specific items clustered into one component, which was also comparable with the QUOTE-Rheumatic-Patients and QUOTE-Elderly.8,14 Four per cent of the patients missed an aspect regarding “follow up visits” in the questionnaire. Future research should indicate whether this missing aspect is of additional value for the questionnaire. The questionnaire as developed at UHM and UHG was shown to be applicable to the REH although four aspects were not included in the QUOTE-cataract at UHG. These aspects seemed to be institute specific for UHM and REH. One should recognise that subtle differences in communities require differences in assessment.22 Therefore, it is recommended that validation of the QUOTE-cataract is carried out at each specific setting and location before its administration. Since process and structure related quality of care (for example, patient education and counselling) have been proved to be highly correlated with patient satisfaction,1 we believe that the QUOTE-cataract may be an adequate instrument to measure patient satisfaction after cataract surgery in a more reliable and valid way.
Anne Lems and Maria Dolders, MHSc, are acknowledged for their contribution to the field work. The authors wish to thank Christine Markham, PhD, for helpful linguistic comments on this manuscript.
The QUOTE-cataract questionnaire is available upon request.
Series editors: G and M Brown
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.