Family-wise Error Rate

Authors
Affiliations

Doctor of Physical Therapy

B.S. in Kinesiology

Doctor of Physical Therapy

B.A. in Neuroscience

AKA
  • Per-comparison error rate
  • Per-experiment error rate

Family-wise error rate refers to the rate or probability of committing one type 1 error when performing a family of comparisons.

The more hypotheses and comparisons a researcher performs, the higher chance of at least 1 type 1 error occurring in one of the variables compared1.

Note

“For studies that test multiple hypotheses or make multiple comparisons, the probability of at least 1 Type I error (family-wise error rate; FWER) increases as the number of hypotheses/comparisons increase”1.

Uncorrected for Family-Wise Error

Without correcting for family-wise error rate, the significance level refers to “The chance that random sampling would lead this particular comparison to an incorrect conclusion that the difference is statistically significant when this particular null hypothesis is true”2.

Corrected for Family-Wise Error

When taking into account family-wise error rate, one is looking at multiple comparisons and determining the significance level as the chance of obtaining one or more statistically significant conclusions if all of the null hypotheses in the family are actually true2.

Importance

The purpose of family-wise error rate is to use a stricter threshold to define significance and reduce type 1 error2. When using the standard significance (\(\alpha = 0.05\)) and all the null hypotheses are true, then we want \(\leq 95\%\) chance of obtaining zero statistically significant results and a 5% chance of obtaining one or more statistically significant results2. That 5% chance applies to the entire family of comparisons performed in the experiment, so it is called a familywise error rate or the per-experiment error rate2.

To organize

When each comparison is made individually without any correction for multiple comparisons, the traditional 5% significance level applies to each individual comparison2.

What is a family of comparisons

What is a family of comparisons? What exactly is a family of related comparisons? Usually, a family consists of all the comparisons in one experiment or all the comparisons in one major part of an experiment. That definition leaves lots of room for ambiguity. When reading about results corrected for multiple comparisons, ask about how the investigators defined the family of comparisons.2

Controlling Family-wise error

3

One can use a Bonferroni Correction to control for family-wise error2.

Supplemental Material

References

1.
Nicholson KJ, Sherman M, Divi SN, Bowles DR, Vaccaro AR. The Role of Family-wise Error Rate in Determining Statistical Significance. Clinical Spine Surgery. 2022;35(5):222-223. doi:10.1097/BSD.0000000000001287
2.
Motulsky H. Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking. 4th ed. Oxford University Press; 2018.
3.
Nichols T, Hayasaka S. Controlling the familywise error rate in functional neuroimaging: A comparative review. Statistical Methods in Medical Research. 2003;12(5):419-446. doi:10.1191/0962280203sm341ra

Citation

For attribution, please cite this work as: