Type of Online Panels
- Vast majority not constructed using probability based recruitment
- Offers to join
- River sampling
Total Survey Error
- Coverage error – major factor when goal is to represent general population
- Approx 1/3rd Us adult population does not use internet on regular basis.
- Under coverage
- High level of non response at various stages of building a non-probability panel and delivering respondents to individual studies
- Major differences between surveys using non probability panels and traditional methods (usually phone) à difficult to determine whether mode of administration or sample bias is greater cause of differences.
- Studies suggest probability sampling still more accurate
Adjustments to Reduce Bias
- Standard demographic weighting
- Simple purposive sampling that uses known information about panel members to generate demographically balanced samples
- Standard quota sampling
- Propensity models in post stratification adjustments à augment standard demographic weighting with attitudinal or behavioural measures thought to be predictors of bias
Concerns about Panel Quality
- Industry response
o Guidelines & standards
o Validating panelists à remove duplicates etc
o Research to understand drivers of panelist behaviours / design techniques to reduce impact of those behaviours on survey results.
Background & Purpose of Report
- Types of panels
o Probability based methods
o Non probability approach / volunteer online panels
Overview
- First challenge for all survey modes à development of sample frame
- Online panels have become popular solution to sample frame problem.
- General population panel à serves as a frame from which samples are drawn to meet specific needs of particular studies.
- Census balanced samples
- Specialty panel
- Proprietary panel
- Targeted samples
Non Probability Volunteer Online Panels
- Lower cost
- Faster response
- Ability to build targeted samples of people who would be low incidence in a general population sample
- Five major areas of activity
o Recruitment of members
§ Motivations
· Contingent incentive
· Self expression
· Fun
· Social comparison
· convenience
o Joining procedures and profiling
§ Double optin process
§ Validation procedures
o Specific study sampling
§ Simple random samples from panels are rare à tendency to be highly skewed to particular demographics
§ Purposive sampling
o Incentive programs
o Panel maintenance
Probability Based Recruitment
- Dutch Telepanel – 1986
River Sampling
- Intercept interviewing / real time sampling
- May be on rise as researchers seek larger / more diverse sample pools and less frequently surveyed respondents than provided by online panels
Errors of Non Observation in Online Panel Surveys
- Coverage
- Non response
- More severe with online panels than other types of surveys
- All sampling frames have features which can affect quality
o Under coverage
o Multiple mappings of frame to population
§ Eg, multiple people per home phone line
o Duplication
§ eg, one person, multiple phone numbers
Online Panel Surveys, Frames, and Coverage Issues
- volunteer panels do not attempt to build complete sampling frame
- notion of sample frame is skipped
- instead focuses on recruitment and sampling steps.
- Common evaluative criterion of volunteer panel is not full coverage of household population, but sufficient diversity on attributes related to type of surveys supported by panel.
- Online panels can repeatedly sample
Unit Non Response and Non Response Error
- Failure to measure a unit in a sample
- Person selected for a sample does not respond to survey
- Vs item non response à respondent skips a question
- 4 stages in volunteer panels where non response can be a issue
o Recruitment
§ No way of knowing anything precise about size / nature of non response at this stage
§ If online panel members belonging to under represented groups are similar to group members who are not in panel, the risk of bias is diminished under an appropriate adjustment procedure
· Within group homogeneity may be a poor assumption.
o Joining and profiling
§ Just over 6% of those who click thru a banner advert to panel registration page eventually become member.
o Specific study sampling
§ Reasons why member may not participate
· Lack of interest / survey length / heavy volume of survey invitations
· Failure to qualify / not completing within require time period
· Technical problems
§ Address differential non-response
· Disproportionately higher rate of group selected.
· Pre emptive differential non response adjustment
· No guarantee that non response error eliminated / reduced
o Panel maintenance
§ Forced or normal attrition
§ Forced turnover is not a form of non response
- Response metrics
o Recruitment is constant on-going endeavour
- Coverage errors versus non response bias
o Given absence of sampling frame in online panels, conceptual difference between coverage errors and non response errors gets blurred.
- Measurement error in online panels
o Understand how and why people think, feel and act.
o Measurement error is defined as difference between observed response and underlying true response
§ How concepts are measured
· The questions and the answers
· Questionnaire design effects
§ Mode of interview
§ Respondents
§ Interviewers
o Mode effects
§ Two hypotheses about possible impact of shifting from one mode to another
· Social desirability
· Satisficing
§ Primacy
· Tendency for respondents to select answers offered at beginning of list
§ Recency
· Select answers from last offered
§ Concurrent – predictive validity
o Respondent effects
§ Cognitive capabilities of panel members
§ Motivation to participate
§ Panel conditioning
· Taking too many surveys
§ Topic interest and experience
- Sample adjustments to reduce error and bias
o Purposive sampling
§ Non random selection technique
§ Goal – sample representative of defined target population
§ Anders Kiaer – 19th century
§ Quota sampling
§ Census balance sample
o Model based methods
§ Small area estimation
§ Epidemiological studies
o Post survey adjustment
§ Weighting techniques
- Panel data Cleaning
-
No comments:
Post a Comment