Questionnaire Design- How To Plan, Structure and Write Survey Material For Effective Market Research.
Second edition
Ian Brace
U 658.83 B796Q 2008
Monday, January 31, 2011
Questionnaire Design: introduction & Chapter 1
Introduction
- Obtaining the best answers
- Data we collect rarely completely accurate
o Ask to recall events that are often trivial, such as breakfast cereals they bought
o Analyse and report their emotions and feelings about issues they have never consciously considered, such as feelings about different brands of paint.
o Even if they can recognise their feelings, can they articulate them.
o Our own instruments are often blunt and rarely capable of assessing what is true & accurate.
- Why do we need a questionnaire
- Questionnaire – medium of remote conversation between researcher and respondent.
- Remoteness
o Lack of interaction
Objectives in writing a questionnaire
- Questionnaires in the survey process
o Bra example
- Stakeholders in the questionnaire
- Objectives of the study
- Relating research objectives to business objectives
- Relating the questionnaire to the research objectives
o Eg paired comparison à will require questions to compare preference between products
- Recruitment questionnaires
- Collecting unbiased and accurate data
o Problems within questionnaire
§ Ambiguity
§ Order effect between questions
§ Order effect within questions
§ Inadequate response codes
· Do you like pizza : yes / no
o Respondent may wish to qualify answer : homemade or purchased.
· How often do you visit cinema
§ More than once a week
§ Once a week
§ Once a month
§ Once every three months
o What about respondent who went twice last week, and not at all in the three prior months.
§ Wrong question asked due to poor routing
o Problems with interview
§ Questions asked inaccurately by interviewer
· Ie paraphrase
· Poorly written, eg too many sub clauses
§ Misunderstanding
· vocab
§ Inaccurate record
· Précis to keep conversation flowing
§ Boredom / fatigue
· Lots of rating questions
o Pattern of responses that bears no/little relationship to their actual answers
· On-line drop out p18
§ Inaccuracy of memory
· Diary
· Minority behaviour tends to be unreported
· Time / telescoping
§ Asking respondents to describe attitudes on subjects which they hold no conscious attitude
· Studies have shown data reported are more stable where respondents are not given time to think about their attitudes
§ Lying
§ Impress
Sunday, January 30, 2011
Topic 6 : Coding & Cleaning Survey Data
Data Preparation Steps
- Interviewing / self completion
- Receiving completed questionnaires
- Validating
- Checking and editing
- Coding
- Data entry
- Verifying
Processing Paper Questionnaire
- Control register
- Outcome
o Completed interview
o Refusal
o Did not speak English
o Ineligible
o No contact
o In phone surveys: dead number / fax / engaged
- Questionnaire identification
Validating
- Determining when questionnaire is suitable to be included in analysis
o Right person completed
o Was interview conducted the right way
§ Validate 10%
§ Monitor real time
§ Call back
Checking & Editing
- Editing – visual check – acceptable for data processing
- Issues
o Incomplete / pages missing
o Pattern of responses may show respondent did not understand instructions / not followed correctly
o Responses may show little variance, eg – all questions rated 4 on likert scale
- Actions
o Check filter / skip / branch questions have been followed
o All relevant questions followed
o Deal with incomplete questionnaires
o Real with irregularities, ie, aged 13 with Phd
è Create new response category : information not provided
è Best guess
Coding
- Pre-coded vs open ended
- Open ended may be coded by researcher
- Expand other category
- Codebook
o Exact question
o Variable name
o Data type
o Explanation of valid codes
o Definition of missing values
o Special instructions
- Issues in coding
o More codes, greater detail
o Fewer codes, easier analysis
- How to code an open response question
- How to code demographic questions
- Coding others
- Data entry
Verification
- Setting data entry controls
- Manually scanning data
- Re-entry
- Computer based verification
- Changing data values (version control)
Saturday, January 29, 2011
Modern Measurement: Modern Scaling with Item Response Theory : Chapter 10
Introductory Description of IRT
- What is IRT
o Psychologically based theory of mental measurement that specifies information about latent traits and the characteristics of stimuli used to represent them.
o IRT statistics are not non-parametric
- Relation of IRT to CTT
- Cautionary note on studying IRT
o IRT : theory about latencies and the way they can be estimated
IRT and Invariant Measurement for Items and Persons
- Problem of lack of an independent scale in CTT
o Difficulty in comparing "low esteem" in one test vs another test.
o No common zero point
o Converting scores to z scores does not solve problem – then you only have scores expressed in same metric
- Group Dependent Items and Item Dependent Groups
o Left with relative comparisons
o Eg, test on history – what measurement best represents difficulty of test à depends on group who takes test à eg, primary school vs college
o Difficulty value is group dependent
o Reference group
o Measuring examinee ability is item dependent
- IRT as item and person invariant measurement
- Notion of invariant measurement
o Invariance is an estimable concept [GW – not sure I understood or agreed with this section]
Introduction to IRT Models
- Some commonly used IRT Models
- Models are usually identified by number of characteristics they estimate about a test's stimuli
o One – parameter
o Two / more
- Most popular models
o One-parameter
§ Only item difficulty is estimated
o Two parameter
§ Estimates separate difficulty and discrimination parameters for each item
o Three parameter
§ Includes examinee's probability of guessing or pseudochance
Assumptions
- Centrality of assumptions to IRT
- Unidimensionalty of items and tests
o Given test item or exercise is directly targeted at single cognitive process, and in theory it fills that latent space completely
- Local independence
o Examinees response to a given specific measure reflects an independent and autonomous reference to a latent trait in cognition.
o Examinee responds to stimulus of test item or stimulus, also approaches the stimulus without also thinking about other items or exercises.
o Degree of learning as more items are encountered
- Item characteristic curve
o Defined characteristics of test stimuli are reliably estimable functions
- Certainty of response
o Optimal performance
ICC and IRC
- Specifying ICCs generally
- Inflection point
- Scales allow a trace line to describe functional relationship between characteristics of an item and the trait level of examinee
IRT Models
- Likelihood function
o Examinee of particular ability level has certain probability of getting an item correct
o .likelihood function is joint probability of getting several items correct or incorrect
o More on working with the log scale
o The two parameter model
The one-parameter IRT model and Rasch
- The Rasch Model
Other IRT Models
- Nominal and graded response models for polytomous items
- Richly cognitive models
Estimating Item and Ability Parameters
- Iterative estimation procedures
- Developing priors
- Test information function
- Some estimation procedures
Computer Programs Available for parameter estimation
Brief History and major Contributors to IRT
Friday, January 28, 2011
Modern Measurement: Chapter 9 : Performance Related Measures
A Psychometric Perspective on Performance Assessment
- Terminology of Performance Assessment
- Defining Performance Assessment
- Examples of PA
Characteristics of Performance Assessments
- Three common characteristics
o Require examinees to respond to stimulus in a manner other than selecting one choice from presented alternatives
o Intended to gauge sophisticated thinking skills
o To score them, someone must inspect response and apply evaluative judgment
Organizing Performance Assessments
- Two rough categories of performance assessment
o Task centered
o Construct centered
- A PA taxonomy
o Type of reasoning competency employed
o Nature of cognitive continuum employed
o Kind of response yielded.
Special Types of Performance Assessment
- Authentic assessment as performance assessment
- Rather than use proxy format
- Complex performance assessments
Validity Issues
- Importance and difficulty in amassing validity evidence for Pas
- PA Validity evaluation as interpretative argument
- Construct evidence for Pas
Essays as Performance Assessments
GW Comment / Question
- Is there a survey equivalent to Performance Assessment – focus group ?
Thursday, January 27, 2011
Modern Measurement – Chapter 8 : Constructing Items and Exercises for Tests
Importance of Test Items and Exercises to Measurement
- Role of items in tests
o We employ items and exercises as sole source to excite mental processes that evoke a measured response – the raw info we evaluate.
- Difficulty in producing meritorious items and exercises
o Requires good writing, which is difficult
- Raw material needed for writing good test items
- Current sources for item preparation
Theory of Test Item Development
Practicalities: Item Types and Their Classification
- Examples
o Multiple choice
o True / false
o Completion
o Likert type statements
o Essay
o Performances
- Types
o Selected responses
o Constructed responses
- Definition of Test Items and Exercises
- Anatomy of an item
- Test Item Nomenclature
o Avoid using word question, as this implies an interrogative statement , many items and exercises are not so worded.
o Terms
§ Direction
§ Graphic
§ Text
§ Stem
§ Distractors
§ Correct Response
o Testlets as Item Format
§ Several questions share common info
Multiple Choice Items and Sophisticated Thinking Skills
Characteristics of Meritorious Test Items and Exercises
- Be congruent with key objectives or psychological constructs
- Have clearly defined key objectives or psychological constructs
- Contribute minimally to error in measurement
- Be presented in a format that is suitable to test goals
- Meet specified technical specifications
- Be well written and follow prescribed editorial standards
- Satisfy ethical and legal concerns
Techniques for Item Writing
- Considerations in Item Development
- Consolidated approach to test development
- Test content specifications
- Congruence of item or exercise to the specification
- More consideration for items
- Case against Lists of Dos and Don't's
Automatic Generation of Test Items and Exercises
- What is item generation
- Linguistic roots of item generation
GW Comments / Questions :
- How different is it preparing a test item vs a questionnaire question
Wednesday, January 26, 2011
Topic 4 : Developing a Questionnaire
What is a questionnaire
- Aim : collect accurate data in a consistent format
Influences on Questionnaire Design
- Purpose of research
o Until we have a clear understanding of purpose of research, no point in thinking about questions
- Who Will be answering the questionnaire
o Style and complexity of language
- How Will Questionnaire Be Administered
o Self completion vs computer based
General Format of Questionnaire
- logical sequence
o simple then complex
o general then specific
o neutral before personal
o "flows" from one topic to another
o Vs omnibus
Basic Structure of Questionnaire
- Title
- Name of sponsoring organisation
- Introduction / purpose & use of survey / confidentiality
- Screening questions
- Main questions
- Verification (respondent contact details)
- Thank you
- Authentication : interviewer name etc / detail of interview
What can we measure in a questionnaire
- Behavior
- Beliefs – what people think is true
- Knowledge – accuracy of beliefs
- Attitudes – what people think is desirable
- Attributes – respondent's characteristics
Behavioral Questions
- Did respondent exhibit behavior
- Frequency
- Time elapsed since
- Intended behavior
Measuring Knowledge & Beliefs
Measuring Attitudes
- What is desirable
- Conviction
- Context / seasonality
- Measured by scale
Recording Attributes
- Descriptive information
- Analyse sub-groups
- Define quotas
- Determine if sample is representative of population
Response Formats
- open ended
- pre-coded
- uncoded open ended – respondent free to provide any answer
- precoded open ended – respondent provides answer of choice, but interviewer then codes it
- pre-coded – respondent has to select choice
o one answer : discrete / single response question
o multiple response
- Open ended
o Analysed qualitatively as anecdote
o Quantitatively – percentage who gave certain response (particularly where number of response options too large to precode)
o Response options unknown
o Get general feelings
o Get reasons for opinions
- Precode
- Which format ?
o Kind of info sought
o Whether researcher wants to pre-condition respondent
o Prior knowledge of possible responses
o Ease of processing info
o Available resources.
o Info potentially available from respondents
o Level of accuracy required
o Sensitivity of questions
o Experience of respondents.
Question Wording Rules
- Set of response options for precoded questions must be exhaustive
- Response categories to be mutually exclusive
- Scale should be unidimensional
o Scale should only measure single concept.
Structuring a Questionnaire
- Logical order
- Apart from primary questions
o Secondary questions – consistency or reliability checks
o Tertiary questions – breathers after sensitive questions
o Probes – gather details : what else
o Direct & indirect questions
§ Direct – about respondent
§ Indirect – info about respondent by asking about other
o Suggestive – assume respondent has particular view on issue and these questions attempt to confirm that view
o Filter & contingency questions – select certain groups of respondents
Minimizing Respondent Burden
- Relevance
- Language used: conversational / clear
- Question length : single issue
- Avoid leading questions
- Dealing with sensitive issues
o Casual approach
o Numbered cards : which one of these statements applies to you
o Everybody approach
o Other people approach
General Samples and Population Means: Chapter 10: Statistics : A Bayesian Perspective
Chapters 6 – 9 of Statistics: A Bayesian Perspective dealt with making inferences about proportions. Chapter 10, and the 2 following chapters deals with inferences about general types of observations. Chapter 10 looks at one sample and one population.
The book approaches inferences about one population by using density both within models and across models. The calculation methods assume that the population of interest is normally distributed, and that as sample size increases, sample proportions tend to population proportions.
As has been the case with previous chapters, the book describes a "spreadsheet" like approach to calculation, and then follows up with a density based calculation method. Chapter 10 looks at the spreadsheet approach for calculating posterior probability based on one sample / population.
This example is about a study of piglet weight gains. The sample size is 100.
The components of the spreadsheet model are:
Model: in this case, there are 1,000 models for the possible weights; from 0 kgs to 100 kgs, in 0.1 kg steps.
Z-score: because we are assuming the population is normally distributed, we calculate the z score using the formula z = √ (n) * (mean – model) / std dev
Likelihood: e^ -(z^2)/2 - ie formula for height of normal density
P (Model): Prior probability. In this example, a flat prior has been used.
Product: Likelihood * Prior
P (Model | D ): Posterior probability
n | 100 | ||||
x | 30 | ||||
std dev | 10 | ||||
e | 2.718282 | ||||
Model | z-score | Likelihood | P (Model) | Product | P (Model | D) |
30.00 | 0.00 | 1.00000000 | 0.000999 | 0.00099900 | 0.03989423 |
30.10 | -0.10 | 0.99501248 | 0.000999 | 0.00099402 | 0.03969525 |
30.20 | -0.20 | 0.98019867 | 0.000999 | 0.00097922 | 0.03910427 |
30.30 | -0.30 | 0.95599748 | 0.000999 | 0.00095504 | 0.03813878 |
30.40 | -0.40 | 0.92311635 | 0.000999 | 0.00092219 | 0.03682701 |
30.50 | -0.50 | 0.88249690 | 0.000999 | 0.00088162 | 0.03520653 |
30.60 | -0.60 | 0.83527021 | 0.000999 | 0.00083444 | 0.03332246 |
30.70 | -0.70 | 0.78270454 | 0.000999 | 0.00078192 | 0.03122539 |
30.80 | -0.80 | 0.72614904 | 0.000999 | 0.00072542 | 0.02896916 |
30.90 | -0.90 | 0.66697681 | 0.000999 | 0.00066631 | 0.02660852 |
31.00 | -1.00 | 0.60653066 | 0.000999 | 0.00060592 | 0.02419707 |
The table is an extract of the complete model set. The extract is centred around those models that are likely.
The models around 30 are the most likely. The posterior probability that the correct model is 30.0 is quite small, as there are many models around 30 that could account for the weight gains observed.
Comments on analysis
- Prior probability is flat; in reality, it is not. Can be sure, for example that pigs will gain at least some weight, but unlikely they will gain 90 pounds in 20 days.
- Assuming population is normal leads to rich theory in which calculations are not difficult.
- Normal densities not always appropriate.
- Smooth within models as well smooth across models.
- Law of large numbers à as sample size increases, sample proportions will increase and tend to population proportions
- Likelihood : height of density at observed data point.
-
Subscribe to:
Posts (Atom)