“Alternatives to null hypothesis significance testing”

"Alternatives to null hypothesis significance testing"

This article (extended reading – week 2) refers to the debate criticising the null hypothesis significance testing (NHST) , and offers some alternatives.

One comment I've come accross in reading various articles on the issue is that the term "significant difference" implies a meaningful or important difference. That's not the case; the term significant is just being used in a specific and precise way within statistics, and merely refers to where , for example, the observed F value lies in reference to the critical value for accepting or rejecting a null hypothesis.

The following article

http://www.stat.columbia.edu/~cook/movabletype/mlm/kaul.pdf

discusses clinical significance as opposed to statistical difference, and introduces the term "minimum clinically important difference"

This seems to be the same as the alternative to NHST that Murphy, Myors and Wolach hilight in their book " Statistical Power Analysis – A Simple & General Model for Traditional & Modern Hypothesis Tests (3rd ed – copy in Swin library)

Murphy et al make the point that in most cases, the null hypothesis is almost always to be rejected, if one's research is successful. In addition, the null hypothesis is almost always a nill hypothesis, that there is no difference, with the alternative being there is a difference.

The other point they make is that with a sufficently sensitive study (ie large enough sample), the statistical power of the traditional null hypothesis can reach 1.
Their suggested alternative is "minimum effect hypothesis test".

In this case the researcher needs to make a call on what is the minimum effect they want to detect. That call will be informed by theory, literature search, etc.

On page 90, they provide an example of research which examines suitability for employment and drug usage. They observe that "although significantly different from zero, it is clear the relationship between these tests and job suitability is so weak the variables can be treated as virtually independent".

When I first read Murphy et al I didn't find it interesting, but on re-reading, found it had a lot of interesting things to say.

Overall, I found the debate about the usefulness of NHST interesting, and the alternatives worth considering.