Wednesday, November 17, 2010

Statistical significance is not the same as significance

There is a lot of confusion among scientists regarding the appropriate use of statistics. A recent short letter on statistics published in the Canadian Medical Association Journal (CMAJ) helps illustrate this problem. CMAJ does not take out a copyright on the letters so we're reproducing it here, although you can access it directly from their website.

Statistical Significance is not the same as Significance

The recent interesting and beneficial article by Ware et al. [1] has been followed up with a response letter regarding the use of statistical testing [2]. This follow-up letter correctly points out that statements such as “nonsignificant degrees of relief” and “higher but not significant”, are both common in the medical literature and should be avoided [2].

I agree with the points made by Cragg [2], however, I feel that the clinical readership could benefit from an even simpler explanation of why statements such as “higher but not significant” should be avoided.

The word ‘significant’ is qualitative. Any individual researcher or clinician is able to decide for themselves what constitutes a ‘significant’ effect. The word ‘significance’ is not synonymous with the term ‘statistical significance’. Statistical significance is typically defined as a p-value below 0.05, which means that there is a less than 5% probability that the observed data occurred due to random chance. Experimental data that achieves ‘statistical significance’ would not necessarily be qualitatively described as significant. Furthermore, experiments with very few samples may yield a qualitatively significant effect that does not achieve statistical significance.

Thus, in the literature, the words ‘statistical significance’ should not be replaced with the word ‘significance’ as it can lead to confusion due to the different meanings of the terms.

Jacob Levman, PhD (Medical Biophysics, University of Toronto)

[1] Ware et al., Canadian Medical Association Journal (2010) 182:E694-701E.

[2] Cragg, Canadian Medical Association Journal (2010) Reply to 182:E694-701E.