Introduction
- survey responses – sensitive to exact wording, format and placement of questions asked.
- factual / non factual
o factual component overlaid with evaluation
§ how is your health now
§ low level of correlation between perception and fact.
- validity studies
o possible with factual only
- methods effect
o artifact of the set of measuring instruments employed
- consistency vs validity
o split ballot experiments
- possible influences on responses
o factual
§ definition
§ comprehension
§ memory
§ social desirability
o non factual
§ issues of balance
§ offer of middle alternatives
§ order of presentation of alternatives
Question Effects with Factual Questions
- precise definition of fact to be collected à apparently marginal changes in def can have profound effects
o unemployment / labor force
o definition of a room
- respondents need to comprehend question
o meaning of week day
o meaning of proportion
- recall
o length of recall period
o salience
o choice of appropriate [time] reference period
o bounded recall
o minimise memory errors
§ use of records
§ aided recall techniques
§ diaries
- social desirability bias
o randomized response techniques
§ any gain in bias reduction has to be offset with sizable increase in sampling error
§ hampers analysis of relationship between threatening question and other variables
- long question vs short question
o keep it short vs keep it simple
o difficulties from long questions probably derive from their complexity rather than their length per se
- instructions
- use of feedback
- securing respondent feedback
Question Effects with Non factual Questions
- conceptualization of construct to be measured
- don't worry about marginal distribution of answers
- look for some form of correlational analysis
- treatment of don't knows
o explicit don't know category
o Specifically ask – do you have an opinion on ……
- open or closed questions
o open
§ responses nominal in nature and sizable in number
- use of balance
- acquiescence
- middle alternatives
- order of alternatives
General Question Effects
- effects noted for one type of question can apply to other type – ie sensitivity
- question re income
- presence of other questions in questionnaire / relative position
o question order / context
- example found of order effect
o general question on issue with specific question on same issue
o position of response on long lists
o crime survey – inclusion of attitude questions – increased reported victimization rates
§ attitude question may have stimulated respondent memory / awareness
Concluding Remarks
- survey questioning not a precision tool
- need to understand nature of context effects
- strong defence against artifacts described in this paper – use multiple questions / contexts and modes of research
- by tieing an important concept to at least a few items that differ among themselves in form, wording, and context, investigator is unlikely to be trapped into mistaking a response artifact for a substantive finding.
Discussion
- tacit support for view that since marginal results are difficult to measure, the concentration on comparisons is acceptable
- surveys need to derive valid methods to measure proportion supporting particular issue
è comment by Hedges
o much could be learnt from more open discussion and criticism of questions, even in the absence of specific experiments
o implicit assumption in literature that in absence of hard evidence there is little point in speculating or reasoning about question effects because any opinion is likely to be as good as any other
è [GW comment : why non prob surveys can provide valuable info]
- this raises the question of whether we model bias in the mean bought about by inaccurate measurement and / or bias in measures of relationships brought about by correlated errors, which may be a more appropriate distinction than the std one proposed in the paper between "factual" and "opinion" questions
- a lot of survey research is concerned with breaking down concepts into their component parts and asking questions on each one of them
o income
o takes 20 or 30 questions to actually disentangle it
- example of chronic sickness and long term incapacity
- questioning problems
o imprecision
§ will lead individual respondents applying their own interpretation
o method effects
o respondent task problems
§ eg, memory / recall
o asking different questions
- donkey vote
- using electoral research à position on ballet paper
- questions and responses
- should be : stimuli and responses
- long questions à give people a chance to think and you get different replies
No comments:
Post a Comment