Statistically Significant vs. Practically Significant in SMT Data Collection

Posted on by

Folks,

Let’s assume your company has decided that transfer efficiency (TE) is the key metric in determining solder paste quality. Transfer efficiency is the ratio of the volume of the solder paste deposit divided by the volume of the stencil aperture. While you agree that TE is an important metric, you are a little troubled with the recent results in a solder paste evaluation. Two out of 10 pastes are fighting for the top spot and it looks like TE will be the deciding metric. Paste A had a TE of 99.5% and Paste B had a TE of 99%. So management wants to go with paste A. You are troubled because paste A has a poor response-to-pause. If it is left on the stencil for 15 minutes or more the first print must be discarded. This weakness may result in 30 minutes or so of lost production time in a 3-shift operation.

However, the TE test results showed that the TE of paste A was statistically significantly better than paste B. You think about this situation and something doesn’t make sense — 5% and 99% are quite close.

You dust off your statistics textbook and review hypothesis testing. Then it hits you, with very large sample sizes, means that are closer and closer together can be statistically significantly different.

The data show that paste A has a mean of 99.5% and a standard deviation of 10%, whereas paste B has a mean of 99% and also a standard deviation of 10%. The sample sizes were 10,000 samples each. These large sample sizes are important in the analysis. The standard error of the mean (SEM) is used to compare means in a hypothesis test. SEM is defined as the standard deviation (s) divided by the square root of the sample size (n):

So as the sample size increases, the SEM becomes smaller or in statistics lingo “tighter.” With very large sample sizes, this tightness enables the ability to distinguish statistically between means that are closer and closer together. This situation was not a concern with sample sizes of less than 100, however with the modern solder paste volume scanning systems of today, sample sizes greater than 1000 are common.

Figure 1 shows the expected sampling distribution of the mean for samples with a TE of 99.5% and 99.0% and a sample size of 100, both have a standard deviation of 10%. Note that to your eye you do not see much difference. However, with the means and standard deviations the same and sample sizes of 10,000 the sampling distributions of the mean are clearly different in Figure 2.

The reality though, is that there is no difference in the results in Figure 1 and 2. The tiny difference in the means (0.5%) may be statistically significant with a sample size of 10,000, but is it practically significant? Would this small difference really matter in a production environment? Almost certainly not.

Figure 1. Sampling distribution of the mean for a sample size of 100.
Figure 2 Sampling distribution of the mean for a sample size of 10,000.

So, with large sample sizes, we need to ask ourselves if the difference is practical. For TE, I think we can be confident that a difference of 0.5% is not practically significant. But, what if the difference was 2% or 5%? Clearly, experiments should be performed to determine at what level a difference is significant.

With the case discussed above, I would much prefer the paste that has a 99.0% TE and a good response-to-pause.

Cheers,

Dr. Ron