Evaluating the Credibility of Interpersonal Comparisons using Survey Experiments and Benchmarks
In this paper I study how individuals use response scales when evaluating themselves in two domains of personal health, and provide an approach for evaluating methods seeking to improve the accuracy of interpersonal comparisons. In particular, I consider methods for adjusting the ordering of individuals based on their ratings of anchoring vignettes. Vignettes are increasingly used in surveys ranging from studies of political corruption to visual impairment.
This study answers (a) how well vignettes improve a researchers ability to determine an ordering of wellness of individuals; and (b) how individual self-evaluations of health status relate to validated measures of health. While the empirical inferences that can be drawn from this study are specific to the sample that is analyzed, this paper provides a set of tools that can be generally applied in other studies that seek to evaluate the interpersonal comparability of survey responses and improve the reliable measurement of attitudes.
This study proposes a survey experiment design to test fundamental assumptions neces- sary for anchoring vignettes to improve our ability to order individuals. A key assumption of anchoring vignettes is that individuals use the same standards for evaluating themselves as they do all the vignettes in the same domain, and that each respondent perceives the vignette in the same manner. In the survey experiment, respondents were asked to evaluate the same stimuli but were randomly assigned either to receive the target stimuli first among a list of vignettes or after being asked to rate a series of other vignettes. This design enables a test of whether individuals have stable standards for evaluating questions. I show that the evaluation of the vignettes do indeed change depending on the order that the vignettes are evaluated, but find that the assumptions of one non-parametric method of using vignettes is consistent with the data.