Article Text
PDF
Abstract
Aim The Visual Disability Assessment (VDA), a questionnaire for measuring the impact of cataract on visual functioning, was developed using classical test theory. Since this approach is limited, our aim was to further investigate the psychometric properties of the VDA using Rasch analysis.
Methods 613 patients from the Flinders Medical Centre cataract surgery self-administered the VDA. Psychometric properties investigated for the overall VDA and each subscale included: measurement of a single construct (unidimensionality), item fit to the construct, reliable discrimination between strata of patient ability (person separation) and targeting of item difficulty to person ability.
Results The VDA discriminated five strata of patient ability. However, seven mobility items constituted a second dimension and formed a valid separate scale. Sequestration of these items resulted in a unidimensional 11-item measure of activity limitation. Both the mobility and activity limitation scales had acceptable person separation and neither contained misfitting items. Targeting was suboptimal for mobility (−2.12 logits) but good for activity limitation (−0.72). The subscales also satisfied the requirements of the Rasch measurement model.
Conclusions The Rasch-scaled VDA effectively measures two separate constructs: mobility and activity limitation (with two subscales). Its good psychometric properties make it suitable for measuring cataract surgery outcomes.
- Cataract
- Rasch analysis
- questionnaire
- visual disability assessment
- lens and zonules
Statistics from Altmetric.com
Introduction
Patient-reported outcomes (or questionnaires) are increasingly being used to track the outcomes of cataract patients.1 2 In recent decades, a number of questionnaires,3 including the Visual Disability Assessment (VDA),4 have been developed specifically for cataract populations and applied in cataract outcomes research.5–7
Like most questionnaires (instruments), the VDA was developed using classical test theory (CTT),4 and while highly validated with this approach,3 CTT has several drawbacks.8 Chiefly, this approach allows limited insight into the psychometric properties of the instrument, and scoring does not provide for interval-level measurement. Scores are simple sums of ordinal values (1, 2, 3, 4) applied to response categories (not at all, a little, quite a bit, a lot) across all questions. This assumes that the quantitative difference between each response category is equal and that each item has the same value. However, neither assumption is valid, which makes the scoring non-linear. This problem can be solved with Rasch analysis, a modern psychometric approach, which transforms ordinal scores into estimates of interval-level measurement.9 This is important, as it allows parametric statistics to be computed from the questionnaire scores facilitating outcome measurement. Furthermore, Rasch analysis provides superior information about questionnaire performance. In particular, Rasch analysis informs whether all items measure a single construct (unidimensionality); this is an essential property of a measurement instrument. If items measure two different constructs, then the score derived from these items is confused by incompatible quantities: imagine a clinical instrument which measured corneal power and intraocular pressure and output these measures as a single number. Rasch analysis also gives insight into the measurement precision, how well the difficulties of the items target the abilities of the patients and performance of items across subgroups of patients. Such insight into instrument performance coupled with the scoring benefits of Rasch analysis have led to it being applied to a number of ophthalmic questionnaires originally developed using CTT.10–13 The results from these studies indicate that Rasch analysis can be applied to examine and then optimise the measurement qualities of such questionnaires.
The VDA4 has not been examined using Rasch analysis. The aim of this study was therefore to use Rasch analysis to re-examine the measurement properties of the VDA. If flawed we aimed to re-engineer the VDA to improve its measurement performance. A secondary aim was to provide spreadsheets that convert raw VDA scores to Rasch-scaled scores, to enable clinicians and researchers unfamiliar with Rasch analysis to utilise its benefits.
Methods
Visual disability assessment
The VDA consists of 18 items that relate to how one's vision interferes with performance of visual tasks (table 1). The items are grouped into three subscales: mobility (seven items), distance/lighting/reading (eight items) and near and related tasks (five items).4 Items 1 and 6 belong to both the distance/lighting/reading subscale and the near and related tasks subscale. Thus, the total number of items across subscales is actually greater than 18 (ie, 20).
Item content of the visual disability assessment
Each item is scored on a four-category rating scale, which consists of not at all (1), a little (2), quite a bit (3) and a lot (4). A total score for the VDA is obtained by adding the scores obtained for each item and then dividing the result by the number of questions answered. Subscale scores are obtained in a similar manner. All items are scored in the same direction with higher scores indicating greater visual disability.
Study population
Participants were 613 patients currently on the waiting list for cataract surgery at the Flinders Medical Centre, Adelaide, South Australia (average waiting period 3–4 months). Participants were mailed the VDA questionnaire for self-administration. Completed questionnaires were returned via a self-addressed and prepaid envelope.
All participants were aged 18 years or older and English-speaking, and had no severe cognitive impairment. The mean age was 73.9 years (SD=9.4), and 55.9% were female. The majority of the participants (60.7%) were awaiting their first cataract surgery. Participants also had coexisting ocular and systemic comorbidities representative of a typical cataract population in Australia. The clinical characteristics of participants are summarised in table 2.
Demographic characteristics of 613 participants who completed the visual disability assessment
Ethical approval was obtained from the Flinders Clinical Research Ethics Committee, and all participants provided informed consent. The study was conducted in accordance with the tenets of the Declaration of Helsinki.
Clinical assessment
Routine clinical assessments were performed prior to referral for cataract extraction surgery. Visual acuity was measured using computerised testing based on logMAR principles, and the screen luminance was 150 cd/m2. As binocular visual acuity is considered representative of real-world ability,14 this was assessed clinically and is reported here.
Rasch analysis
The data were analysed using the Andrich rating scale model15 with Winsteps software (version 3.68).16 The Rasch model is a probabilistic mathematical model which provides estimates of person ability and item difficulty along a common measurement continuum, expressed in log-odd units (logits). A logit is the natural log-odds of a participant being successful at a task versus being unsuccessful. For the VDA, a positive item logit indicates that the item requires a lower level of ability than the average (ie, the item is relatively easier). A positive logit for participants (person ability) suggests that the participant's visual disability is greater than the mean required level of ability for the items (ie, the overall ability required for the tasks is greater than the ability that the participant possesses).
The first step was to examine the performance of the response categories.17 Categories should follow the intended hierarchy; that is, they should demonstrate a stepwise change in ability level from category to category (a little difficulty should represent a higher level of ability than quite a bit of difficulty). If hierarchical ordering is not observed, then categories need to be combined to repair category performance and enable further analyses.
Rasch analysis provides summary statistics in the form of person separation that represents the extent to which the items distinguish between statistically different levels of participant ability. Larger person separation indicates higher precision, meaning more distinct levels of function can be distinguished.18 Person separation >2.0 was considered acceptable, as it indicates that the questionnaire can differentiate between three distinct levels of participant ability (ie, high, medium and low ability).
Fundamental to measurement is the concept of unidimensionality, which occurs when only one construct is represented in the measurement score. Fit statistics are primarily used to indicate unidimensionality. Fit was investigated using the infit mean square (MnSq). The MnSq value is sensitive to unexpected behaviour that affects responses to items near the participant's ability level,19 and represents the observed variance divided by the expected variance. Therefore, the desired MnSq value is 1.0. In the current study, misfitting items were indicated by infit MnSq values outside the range of 0.7 and 1.3 (30% more or less variance than expected).19
Fit statistics alone are inadequate to determine dimensionality. A principal-components analysis (PCA) of the residuals (observed minus expected scores) is performed as confirmation of dimensionality.20 In the present study, we hypothesised that the VDA items represented a single factor, visual disability. We tested this hypothesis by examining the PCA eigenvalues and contrast loading statistics. In the PCA, a high level of variance (>60%) explained by the principal component indicates that there is a very low possibility of finding additional components. Items were considered to load onto a contrast if loadings were higher than 0.3. A contrast was considered to be evidence of multidimensionality if it had the strength of at least two items (as measured by an eigenvalue >2.0), as this is greater than the magnitude seen with random data.16 19
The suitability of question difficulty for the cataract population was examined by inspection of targeting (ie, matching of item difficulty to participant ability). In a well-targeted measure (ie, with a balance of easy and difficult items) the mean locations of the items and persons will be in close proximity to each other (optimal targeting <0.5 logits, good targetting 0.5–1.0 logits). Mistargeting means that either the items are too easy or too difficult for the abilities of the participants.18
Ideally, item difficulty should be comparable across groups, and the scale should operate in the same way irrespective of the group assessed. Differences in item difficulty across participant groups is termed differential item functioning (DIF).21 In the present study, DIF was evaluated for age based on median split (<76 years as younger, ≥76 years as older), gender (male, female), cataract status (first eye, second eye cataract surgery), and ocular, and systemic comorbidity (present, absent). We considered these DIF variables a priori in the present study. In some samples, and particularly large ones, DIF may occur (ie, be statistically significant) but have little practical significance. For these reasons, we defined DIF according to magnitude: <0.50 logits as inconsequential DIF, 0.50–1.0 logits as minimal (but probably inconsequential) DIF and >1.0 logits as notable DIF.10
The measurement properties of an overall (complete) questionnaire do not infer measurement properties of individual subscales. Therefore, the three subscales of the VDA were investigated separately, using the criteria described above. Descriptive analysis was performed using SPSS for Windows (version 15.0; SPSS, Chicago, Illinois), and statistical significance was set at p<0.05.
Results
Analysis of response categories
Response categories were ordered. This indicated that the participants utilised the categories as intended.
Overall performance of the visual disability assessment
Person separation was above the acceptable limit, indicating that the VDA could distinguish reliably between several strata of participant ability (table 3). All items had infit MnSq values within the suggested limits. PCA analysis of the residuals (eigenvalue 2.6 (5.1%) for the first contrast) revealed that seven items loaded positively (>0.3) onto the first contrast, and all these items were related to mobility. This result demonstrated that there was breach of unidimensionality because an additional dimension (mobility) was also being measured by the VDA.
Overall performance of the native version, 11-item activity limitation scale and seven-item mobility scale of the revised visual disability assessment
Based on the above findings, the 18-item VDA was segregated into a seven-item ‘mobility’ scale and 11-item ‘activity limitation’ scale. These scales were then analysed individually.
Mobility scale
Similar to the full version, the seven-item mobility scale had satisfactory person separation (table 3). All items fitted the underlying construct, indicating the scale was unidimensional. PCA of the residuals further supported the unidimensionality. However, compared with the full version, the targeting of item difficulty to participant ability was poor; the ability (ie, level of mobility) of the participants was greater than the average item difficulty. No item showed DIF.
Activity limitation scale
Person separation for the activity limitation scale was satisfactory (table 3). Unidimensionality was evidenced by a lack of item misfit and the PCA of the residuals. Targeting was satisfactory; items closely matched the abilities of the participants (figure 1). Two items demonstrated minimal DIF by gender.
Person-item map of the 11-item activity limitation scale of the revised Visual Disability Assessment. The participants are mapped on the left of the dashed line and those with lower visual disability are located at the bottom of the map. Items are located on the right of the dashed line and harder items are also located at the bottom of the map. Each ‘#’ represents three participants, and each ‘.’ represents one to three.
Analysis of the subscales
Table 4 presents the analysis of the subscales from native version. All three subscales demonstrated adequate performance. The distance/lighting/reading subscale performs optimally, including ideal targeting of item difficulty to person ability. The mobility subscale is not so well targeted to person ability and is identical to the revised mobility scale above. The near and related tasks subscale possessed borderline person separation.
Results of subscale analysis of the visual disability assessment
Conversion of raw scores to Rasch-scaled scores
Since populations vary, it is always best to implement Rasch measurement properties by actually performing Rasch analysis. However, other investigators may wish to use the VDA and also gain the interval scoring benefits of Rasch analysis, without performing Rasch analysis themselves. Therefore, we have provided a series of Excel (Microsoft, Seattle, Washington) spreadsheets which convert raw (ordinal) VDA scores to Rasch measurement estimates. These spreadsheets can be downloaded directly from the journal's website or obtained by contacting the corresponding author.
Discussion
A fundamental problem with the VDA was the lack of unidimensionality that arose from the mobility items measuring a construct distinct from the remaining items. The absence of unidimensionality violates the fundamental requirement for summating a scale.9 Nevertheless, unidimensionality could be restored, but this required that the mobility scale be separated from the activity limitation scale. This revision is consistent with the original aims of the VDA, which were to measure activity limitation and difficulty with mobility in cataract patients.4 The formation of separate measurement of mobility and activity limitation was supported by the results of PCA of residuals, and the fit of items to each Rasch model. This finding was also consistent with previous studies which have found mobility items to form a construct separate from activity limitation in low-vision patients.22
The VDA was also designed to have three subscales, implying that there are three dimensions of visual functioning.4 One of these subscales was identical to the mobility scale identified in this study. The other two subscales (distance/lighting/reading and near and related tasks) both measure within the activity limitation construct. However, it is unclear whether these subscales truly tap different aspects of functioning or simply predict the overall score from the activity limitation scale. What is clear is that all three subscales are valid unidimensional measures which can be used to report cataract surgery outcomes. This finding is important, as measurement validity of the entire questionnaire does not confer validity onto the subscales; this needs to be proven separately.
A number of other instruments designed for measuring visual disability in the cataract patient have been revised using Rasch analysis including the Catquest-9SF,10 the Activities of Daily Vision Scale (ADVS),11 the Cataract Symptom Scale (CSS)12 and the Cataract TyPE Specification (TyPE).23 In all cases, multidimensionality was a problem with the original instrument, but with misfitting items removed, valid measurement was possible except with the ADVS. These instruments are very similar, varying from nine to 12 items in length, and all measure activity limitation, plus the TyPE includes two subscales, and the CSS has a separate mobility scale like the VDA. The main performance difference is in terms of targeting of item difficulty to person ability. The ADVS, CSS and TyPE all suffer from poor targeting.11 12 23 Prior to this study, the only questionnaire to exhibit good targeting was the Catquest-9SF.10 For the VDA, targeting was good for the activity limitation scale but poor for the mobility scale. This probably reflects the fact that cataract patients experience little difficulty with mobility tasks. Therefore, in a cataract population, only the activity limitation scale may be required, unless one were particularly interested in mobility or were working with a less able population. Adding difficult items that target participants with less visual disability could improve the targeting of the 11-item activity limitation scale. However, redeveloping legacy questionnaires is tedious and potentially could fail. Therefore, a superior strategy may be the development of a visual disability item bank and computer-adaptive testing (CAT).24 In CAT, Rasch calibrated items are presented based on the individual's response to previous items, which tailors the presentation of items to the ability of the participant. This approach has been used in other areas of healthcare25 and should now be developed in Ophthalmology.
To conclude, Rasch analysis identified two unidimensional constructs in the VDA: mobility and activity limitation. The revised and improved Rasch-scaled VDA could be used to enhance our understanding of the outcomes of cataract surgery.
References
Footnotes
Funding This research was supported in part by National Health and Medical Research Council (Canberra, Australia) Centre of Clinical Research Excellence Grant 264620. Konrad Pesudovs is supported by National Health and Medical Research Council (Canberra, Australia) Career Development Award 426765.
Competing interests None.
Patient consent Obtained.
Ethics approval Ethics approval was provided by the Flinders Clinical Research Ethics Committee, Flinders Medical Centre, South Australia.
Provenance and peer review Not commissioned; externally peer reviewed.