Controlled Experiments on The Web

 

-          Ron Kohavi et al

-          Data Mining and Knowledge Discovery

 

-          manual (eg OLAP) or machine learning

 

Controlled Experiments

-          simplest controlled experiment à A/B tes

-          Overall Evaluation Criterion

o   scorecard approach – with single metric

o   forces tardeoffs

-          A/A test

-          use OEC components that have inherently lower variability

o   impact on sample size

-          effect of robots on experimental results

 

Analysis

 

-          mine the data

-          speed matters

-          run continuous A/A tests

-          Beware day of week effects

 

 

-          many organisations have strong managers who have strong opinions , but lack data, so we started using the term HiPPO (Highest Paid Persons Opinion) to remind every one that success really depends on users perceptions.