Model significance vs pairwise differences
Original question from a Scientist:
“My ANOVA says the model is not significant, yet some pairwise comparisons are significantly different as evidenced by the gap between their least-significance difference (LSD) bars (which, by the way, is a very helpful feature of Stat-Ease software!). If there is even a single significant difference between any two levels, then wouldn't that create a significant overall model?”

Answer from Stat-Ease Consultant Martin Bezener:
“The two tests are not the same. You should first check the analysis of variance (ANOVA). Then, only if the model is significant (p<0.05), look at pairwise differences on your factorial-effects graph.

Each pairwise test produces a false-alarm rate of 5%. Therefore, with many comparisons, the chance of having at least one false alarm becomes unacceptably high. With multiple responses, the odds of a false positive go up further. The F-test on the overall model also creates a false alarm rate of 5%, However, this encompasses any type of false alarm, so it's typically more conservative.

In your case, with 3 treatments each on 4 lots (a 3x4 multilevel-categoric factorial design), a few combinations look different, but most of them overlap, thus causing the overall model to come out insignificant. It's not a perfect system, so, if, based on your subject-matter knowledge, you believe that one of the pairwise differences may be true (not just a chance occurrence), then do further testing to see if this can be confirmed.”

To see an interaction plot with LSD bars, look over this tutorial example of a multilevel-categorial factorial experiment. In this case (life of batteries made from a number of materials and tested at a variety of temperatures) the model emerges highly significant (p<0.0001). However, the interaction barely achieves significance (0.01<p<0.05), thus some of the LSDs overlap. Nonetheless, this study came to a successful conclusion due to some compelling differences. Check it out!  - Mark
 

On Wednesday February 16, at 2pm CST, I will present a talk on
“DOE for Non-Manufacturing” sponsored by the International Society of Six Sigma Professionals (ISSSP). For an abstract of my presentation and a link for registration, click here.
 

Check out these highly educational presentations coming soon from our world-class DOE experts:

• January 26—“Special Modeling Methods for Non-Ordinary Responses” by Martin Bezener
• February 9—“Designing More Efficient and Effective Experiments for Basic Research” by Shari Kraber
• Feb. 16—“A Crash Course in Mixture Design of Experiments” by Martin Bezener
• Feb. 23—“Evaluating and Exploiting Existing Data” by me (Mark)

You can do no better for quickly advancing your DOE skills than attending a Stat-Ease workshop. Our expert instructors provide you with a lively and extremely informative series of lectures interspersed by valuable hands-on exercises. Enroll early to ensure your spot!

• DOE Crash Course for Experimenters 
  • January 18, 10 AM CST, 1-hour free webinar
• Modern DOE for Process Optimization, 5 half-days online, $945 
  • January 24-28
  • March 14-18
• Mixture Design for Optimal Formulations, 4 half-days online, $945
  • February 14-17
  • April 25-28

See this web page for the complete schedule of upcoming Stat-Ease courses. To enroll in the workshop that suits you best, click Register on that webpage, or click here to contact us.
 

Circulation: Over 4000 worldwide