FAQ
Lack of Fit is Significant: Now What?
Original question from a Research Scientist:
“I generally understand the lack of fit test but I’m struggling to understand how to think through a model that has great statistical properties (very significant model, R^2 predicted >0.9, diagnostics look OK, high adequate precision, etc.) but has a significant lack of fit (p < 0.05). I’ve gone through the different model selection algorithms and most of the models I generate look great except all have a significant lack of fit. I don’t know if this is the correct way to think about it, but this tells me that a significant model can be built that explains most of the data, but the ‘residual’ data unexplained by the model is due to the model not fitting all of the data versus experimental error (pure error used for lack of fit calculation). Am I thinking about that correctly? Or are there other “checks” to do?”
Answer from Stat-Ease Consultant Shari Kraber:
“When I look at client data that exhibits strong lack of fit (LOF) but all other stats are good, the first thing I check is the replicates. If the replicates are very, very similar, then the lack of fit test can easily become significant, even if there are no other modeling problems. The denominator of the test is determined by the difference between the replicates and is simply very small, artificially inflating the LOF test. Maybe the replicates were done all at the same time (like when center points are all run together) causing low variation, or perhaps the center point is a standard setting and therefore the operators are very capable of keeping the process in tight control at that point. In any case, very little variation between the replicates is probably the number one cause of significant lack of fit, in the presence of all other good statistics. In this case I would likely determine that the test is not valid.
“If the replicates are good, then I think of significant lack of fit as telling me that there is more curvature to be explained somewhere in the design space. The next question is - do you care? Where is the model not fitting well? Looking at the 3D plots with the design points showing can help you find the area of the space that has more variation (design points are far away from the surface). Is this a region that you want to predict well? If so, then you may need to add some additional runs in that area to gather more data. If it is in an undesirable area, then don't worry about LOF. It may be that, if the design was a central composite design, the alpha points were set too far out and in order to model the entire space, a cubic model would be required. Do you need that whole space? Or is quadratic sufficient within the area of interest?
“All in all, when any particular statistic is not up to expectations, then using point prediction and the confirmation node becomes extra important. Decide where the optimal settings are and run a few runs at those settings. Make sure they provide the answer you are looking for. Models are approximating the real world, which doesn't follow a polynomial model anyway.”
PS I agree with Shari that if lack of fit stands out as the only problematic statistic, it need not be a ‘show stopper’. If the R^2s adjusted and predicted are positive and adequate precision exceeds the signal-to-noise ratio of 4 (as recommended in the Fit Statistics annotation), press ahead to the diagnostics. Assuming they appear normal, check out the actual lack of fit presented by the Predicted vs Actual plot. It may not be nearly so bad as you might think.
- Mark
|
