Case JMP039: Detergent Cleaning Effectiveness Statistical Modeling – Analysis of Variance Produced by Kevin Potcner, JMP Global Academic Team [email protected] J M P A C A D E M I C C A S E S T U D Y 2 Detergent Cleaning Effectiveness Statistical Modeling – Analysis of Variance Key ideas: This case study requires constructing an ANOVA-based statistical model to explore and describe the effect that multiple factors have on a response, as well as identifying conditions with the most and least impact. Background The effectiveness of a consumer product is of paramount importance, both to the company that manufacturers it and the consumer who uses it. This effectiveness determines the product’s price point, customer satisfaction, and ultimately profitability for the company. One such consumer product used in almost every household is laundry detergent. Detergent manufacturers are constantly working on developing new formulations that can perform better than the current one and be manufactured at a lower cost. The R&D team for one such manufacturer has developed a promising new laundry detergent formulation that can be produced with a 10% cost reduction. To study the effectiveness of this new detergent formulation and to compare its performance to the current formulation, the following experiment was performed and the results stored in a JMP data table. A set of cotton fabric specimens were prepared by being soiled with a dirt-based substance uniformly across the fabric. The test specimens were cut into two sub-specimens. One of them was washed using the current formulation of detergent and the other with the new formulation. After washing, each sub- specimen was measured using a reflective densitometer to obtain a brightness measure. The difference between the two brightness readings was used to create a metric that is a measure of the percent increase in brightness of the new formulation versus the current one. Specifically, the value 0 represents no difference in the brightness readings of the two sub-specimens, a value of 10.0 represents 10% more brightness in the sub-specimen washed with the new formulation compared to the current formulation, and a value of -10.0 represents the sub-specimen washed with the new detergent having a brightness reading 10% less than the one washed with the current formulation. In order to study the effectiveness of the new formulation across a range of washing conditions, all combinations of three water temperatures (Cold, Warm, Hot), two washing times (20 minutes, 40 minutes), and three agitation levels (Low, Med, High) was used. This resulted in 3x2x3=18 experimental treatment combinations. An important principle in designing experiments is to obtain an estimate of experiment error, which is an estimate of the variation that occurs between experimental units receiving the same treatment. The experimenters decided to replicate the experiment so that each treatment combination of Temp, Time, and Agitation was performed twice, resulting in 36 experimental runs, since (3x2x3)x2=36. A diagram of the experiment is shown in Exhibit 1. 3 Exhibit 1 Experimental Design The Task The primary objectives of this experiment are to: 1. Compare the performance between the two formulations quantifying the cleaning effectiveness of the new formulation compared to the current. 2. Determine if the difference in cleaning effectiveness between the two formulations is consistent across all Temperatures, Agitation Levels, and Washing Times or if the difference is dependent upon the specific washing conditions. 3. Determine the specific temperatures, agitation levels, and washing times that result in the greatest difference in cleaning effectiveness between the two formulations. Determine the conditions that result in the least difference. Determine if there any conditions where there is either no difference or the new formulation performs worse. The Data Cleaning Effectiveness_1.jmp Stain Stain type (Dirt) Temp Water temperature (Cold, Warm, Hot) Time Washing time (20 minutes, 40 minutes) Agitation Level of agitation (Low, Med, High) Rep Replicate of each experimental condition (1, 2) %Brightness The percent increase in brightness of the sub-specimen washed using the new formulation of the detergent versus the current formulation 4 Analysis Graphical We begin by visualizing the data. Exhibit 2 is a graph displaying the %Brightness values for all the 3x2x3=18 different experimental conditions (colored dots) along with the average values for the two replicates (shown in the colored lines). Exhibit 2 Scatterplot with Mean Connect Lines To create, Graph>Graph Builder. Select %Brightness as the Y variable, Temp as the X variable, Time as the group variable, and Agitation as the overlay variable. Select Points and Line of Fit in the graph palette. Note:
Any combination of roles for the experimental factors (e.g., X
variable, group variable, and overlay variable) can be used. A good practice is to choose a layout that best communicates the features you wish to convey, such as assigning the X and overlay variables to the factors you think will be affected most. A few features are seen in the graph. The %Brightness values are the highest for the Low Agitation level, while High Agitation has the lowest values. Results for the Med Agitation are between the two but closer to High Agitation. The Cold Temp values have the highest %Brightness values compared to the Warm and Hot Temp. The %Brightness values are also higher for 20-minute wash Time. It is important to refrain from reaching final conclusions just yet without more formal statistical analyses. Some of these differences we see in the graph may be random experimental variation rather than statistically significant. 5 Numerical summary To summarize the experimental results numerically, a table of the mean and range for the 18 experimental conditions is shown in Exhibit 3. Exhibit 3 Tabulate To create, Analyze>Graph Tabulate. Drag the experimental factors into the desired drop zones on the side. Drag Mean and Range into the column drop zones to create the desired table. Drag the %Brightness variable into the center of the table. Note:
You may also use other table configurations. It is a good practice to
choose a layout that best communicates the features you wish to convey. This table provides numerical summaries of the experimental results. Though the graphical display is a much easier way to see and compare the results across the different experimental conditions, it is still very important to generate numerical summaries as they allow us to quantify those results. Analysis of variance model An analysis of variance (ANOVA) model is a standard technique used to compare means across various experimental conditions. Exhibit 4 shows the Effect Summary table from creating an ANOVA model that contains main effects for each of the three factors and the three possible two-way interactions between those factors. 6 Exhibit 4 Effect Summary Table To
create, Analyze>Fit Model. Select %Brightness as the Y variable.
Highlight the three experimental factors (Temp, Time, and Agitation)
and choose Macros>Factorial to Degree. Note: Degree 2 is set as the
default. It will fit a model containing the three main effects for Temp, Time, and Agitation, as well as the three possible two-way interactions (Temp*Time, Temp*Agitation, Time* Agitation). The Effects Summary table displays p-values corresponding to statistical tests evaluating the hypothesis that each of the six terms we added to the model are actually helpful in describing the data. That is, the table demonstrates if there is evidence to conclude that a term is statistically different than zero and thus should be included in the model. The LogWorth value is -log10(p-value). This transformation adjusts the p- values to provide a more appropriate scale for graphing. A value that exceeds 2 is significant at the 0.01 level because -log10(0.01)=2. The standard approach to creating an ANOVA model is to reduce the model such that it only includes terms that are statistically significant, thus the terms that are useful in describing the features in the data. This process of reducing a model begins by examining the most complicated terms first. In this case, it means looking at the three two-way interactions. The p-values for each of these are 0.45182 for Temp*Time; 0.42570 for Time*Agitation; and 0.26322 for Temp*Agitation. They are all quite a bit larger than any standard significance level used (e.g., 0.01, 0.05, 0.10), which indicates that these interaction terms are not helpful at describing the features in the data. An ANOVA model without interactions terms can be interpreted as the effect that each one of the three experimental factors has on the response is similar across the levels of the other factors. Exhibit 5 shows the same Effects Summary table with the three interaction terms removed. Exhibit 5 Effect Summary Table To create, highlight the three interaction terms and choose Remove. The p-values for the three main effects are highly significant (<0.0000 for all). This demonstrates that there is a large amount of statistically significant evidence to indicate that the mean %Brightness values are not equal across the different levels of each factor. 7 Time has only two levels (20 minutes and 40 minutes). The hypothesis being tested in the Effects Summary table is: H0: "#$% '( = "#$% )( HA: "#$% '( ≠ "#$% )( The experimental factor Temp has three levels. The hypothesis being tested in the Effects Summary table is: H0: "%$+ ,-./ = "%$+ 012$ = "%$+ 3-4 HA: "%$+ ,-./ , "%$+ 012$ and "%$+ 3-4 are not all equal Note that the alternative hypothesis HA does not state that “all three means are not equal” but instead, “the three means are not all equal.” The significant result we have indicates only that at least one of the means is different. A further analysis would be needed to determine which ones are different. The experimental factor Agitation also has three levels. The hypothesis being tested in the Effects Summary table is: H0: 56 7-8 = 56 9%/ = 56 3#6: HA: 56 7-8 , 56 9%/ and 56 3#6: are not all equal Similar to the conclusion for the factor Temp, further analysis will be required to determine which ones are different. 8 Multiple comparisons Exhibit 6 displays the results of the multiple comparisons for Temp using Student’s t method. Exhibit 6 Multiple Comparison (Student’s t) To
create, select Estimates>Multiple Comparison under the red triangle
menu at the top of the output. Select the variable Temp and choose All Pairwise Comparisons – Student’s t. Click OK. Then select All Pairwise Differences Connecting Letters under the red triangle next to Student’s t All Pairwise Comparisons heading. A set of statistical tests are conducted for each possible pairwise difference. These hypotheses can be written as: H0: “%$+ ,-./ = “%$+ 012$ HA: “%$+ ,-./ ≠ “%$+ 012$ H0: “%$+ ,-./ = “%$+ 3-4 HA: “%$+ ,-./ ≠ “%$+ 3-4 H0: “%$+ 012$ = “%$+ 3-4 HA: “%$+ 012; ≠ “%$+ 3-4 9 The p-values for each are all statistically significant (<0.0001 for Cold vs. Warm, <0.0001 for Cold vs. Hot, and 0.0111 for Warm vs. Hot), indicating that we have statistical evidence suggesting that the mean %Brightness for each temperature is different than the others. These differences are further described by 95% confidence interval estimates of the mean difference between each comparison. For example, a 95% confidence interval (CI) for (μTemp Cold – μTemp Hot ) is [3.28, 5.16], which is the largest difference of the three. In other words, this CI means that one can have 95% confidence that the mean %Brightness at the Cold Temp is between 3.18 to 5.07 percentage units larger than that of the Hot Temp. The All Pairwise Connecting Letters table is a visual way to display the comparisons that resulted in a statistically significant difference. Factor levels that share a letter are not statistically different. Here, since all the means were determined to be different, none of the Temp levels share the same letter. The All Pairwise Comparison scatterplot is a visual way to display the results of these comparisons. The points are plotted at the coordinates of each pair of means. For example, the point in the lower right (17. 7, 13.5) is the mean value for Cold Temp on the X axis and the mean value for Hot Temp on the Y axis. You can see the label for each by hovering the cursor over the point. The diagonal line represents the place where all of the means would be equal. The confidence interval for each pairwise comparison is shown as the red line extending from each point. If the confidence interval crosses the diagonal line, the pair being compared is not statistically significantly different and is color-coded blue. If the CI does not cross the diagonal, as seen here, the pair being compared is statistically different and is color-coded red. Note that the CI for the comparison between the Cold vs. Hot Temp is furthest from that diagonal line consistent with the CI for that difference being furthest from 0 as we saw earlier. The Warm vs. Hot Temp comparison has a CI closest to the diagonal line. The 95% confidence interval for the difference in those means is [0.305, 2.185] , the closest to 0, and has the largest p-value (0.0111). Technical note: The p-value for the Warm vs. Hot comparison is greater than 0.01, though just barely. If we had chosen 0.01 as the significant level in the hypothesis test (i.e., 99% confidence level), we would not have concluded a statistically significant difference and the confidence interval displayed in the comparisons scatterplot would have crossed the diagonal line and been color-coded blue. This demonstrates that it is not uncommon in practice to find a significant result at one commonly used significance level but not at another, which is why it’s important to be careful not to interpret statistical analysis results as a strict “Yes” or “No” binary decision regarding a hypothesis. Instead it’s best to think of it more as a continuum of evidence that supports or does not support a hypothesis at a certain level of confidence. 10 Exhibit 7 displays the results of the multiple comparisons for Agitation using Student’s t method. Exhibit 7 Multiple Comparison (Student’s t) To create, select Estimates>Multiple Comparison under the red triangle menu at the top of the output. Select the variable Agitation and choose All Pairwise Comparisons – Student’s t. Click OK. Then select All Pairwise Differences Connecting Letters under the red triangle next to the Student’s t All Pairwise Comparisons heading. These results (low p-values and CIs that do not cross the diagonal line) demonstrate that there is statistically significant evidence to conclude the mean %Brightness is different across all three levels of Agitation. Similar to the Hot vs. Warm Temp comparison, the statistical test for the Med vs. High Agitation comparison is not significant at the 99% confidence level but is at the 95% confidence level. 11 ANOVA model To get a better understanding of the ANOVA model that we’ve fit to these data, we can view the equation for the mo
del, as shown in Exhibit 8. Exhibit 8 Predicted Expression To create, select Estimates>Show Predicted Expression under the red triangle at the top of the output. The predicted value for each treatment combination is obtained through this equation. A visual display of the model is typically the best way to “see” what the fitted model is. Exhibit 9 shows that visualization set at the highest predicted response in the first display and at the lowest predicted response in the second. 12 Exhibit 9 Prediction Profiler To
create, select Factor Profiler>Profiler under the red triangle at
the top of the output. Drag the dotted red line to any of the factor settings to see the predicted value along with a 95% confidence interval for the mean response. The predicted value for the Cold Temp, 20 minutes Time, and Low Agitation is 21.5 (the highest predicted value) with a 95% CI for the mean response of [20.6, 22.5]. The predicted value for the Hot Temp, 20 minutes Time, and High Agitation is 10.3 (the lowest predicted value) with a 95% CI for the mean response of [9.4, 11.3]. It is important to remember that %Brightness, the variable being analyzed, is a measure of the percent increase in brightness between one sub-specimen washed with the new formulation versus the other washed with the current formulation. Our analysis has shown that the Cold Temp, 20 minutes Time, and Low Agitation is the combination of conditions in which the new formulation has the most improvement from the current formulation. These data do not allow us to make a conclusion about which washing conditions resulted in the most effective and least effective cleaning for either of the formulations. Custom comparisons It can be informative to make a statistical comparison between specific experimental conditions of interest. For example, an examination of the hypothesis test and a confidence interval estimate for the difference between the two experimental conditions that produced the highest and lowest predicted response is shown in Exhibit 10. 13 Exhibit 10 Multiple Comparison (Student’s t) To create, select Estimates>Multiple Comparison under the red triangle at the top of the output. Choose User-Defined Estimates. Select the experimental conditions Temp=Cold, Time=20, Agitation=Low and click Add Estimates. Select the conditions Temp=Hot, Time=40, Agitation=High and click Add Estimate. Select All Pairwise Comparisons –Student’s t for the Comparison. Click OK Here we see that the difference in the mean response between these two experimental conditions is estimated to be 11.22 with a 95% confidence interval estimate for that difference of [9.69, 12.76]. Model performance and diagnostics When building a statistical model as we’ve done here, it’s important to evaluate how well that model fits the data. Exhibit 11 is one such tool to visualize that fit. Exhibit 11 Actual by Predicted Plot This plot will be displayed if either Effect Screening or Effect Leverage Personality was chosen from within the Fit Model platform.
If not displayed, choose Row Diagnostics>Plot Actual by Predicted
under the red triangle at the top of the top of the output. The actual %Brightness values are plotted on the Y axis and the predicted %Brightness values on the X. The red line corresponds to the actual values being equal to the predicted values. The variation around that line provides a visual of the excess variation remaining in the data beyond the fitted model. The R- squared statistic is a numerical measure of how well the fitted model describes the variation in the data. 14 Here we see that our model accounts for 89% of the total variation in the data with 11% of the variation remaining unexplained. Exhibit 12 is an alternative way to visualize the variation in the data remaining beyond what the model is able to account for. Exhibit 12 A Few Words Describing Exhibit This plot will be displayed if either Effect Screening or Effect Leverage Personality was chosen from within the Fit Model Dialog.
If not displayed, choose Row Diagnostics>Plot Residuals by Predicted
under the red triangle at the top of the output. The predicted %Brightness values are plotted on the X axis, and the residuals (Actual %Brightness: Predicted %Brightness) are plotted on the Y axis. Data points that fall on the horizontal line at 0 are observations where the predicted value is the same as the actual value. Data points above the line are observations where the actual %Brightness value is greater than the predicted value, and data points below the line are observations where the actual value is less than the predicted value. This graph is a convenient way to examine the variation around a fitted model and identify unusual observations regardless of how many terms are in a model and/or how complicated the form of the model is. This graph is also important to consider when building an ANOVA model since it allows us to check one of the assumptions in the inferential techniques we’ve done thus far. The statistical tests and confidence intervals formed from our analysis are based on the assumption that the variance in the data is very similar across all possible treatments studied (i.e., homogeneity of variance), specifically that a common estimate of experimental variation is used to quantify the uncertainty for all the tests and confidence intervals. If there are substantial differences in variation across different experimental conditions, then some tests and confidence intervals are using an experimental error that is too large, while others are based upon an estimate that is too small. Examining this graph indicates that homogeneity of variance is an appropriate assumption. Alternative techniques exist to conduct these types analyses if homogeneity of variance is an issue. In addition, unequal variation across the different experimental conditions may, in fact, be a very interesting discovery revealing important information about the process under study. Another assumption in the inferential techniques performed in an ANOVA model is that these residuals can be well modeled by a normal distribution. Exhibit 13 shows a normal quantile plot of the residuals as a way to evaluate this assumption. 15 Exhibit 13 Normal Quantile Plot of Residuals To create, select Row Diagnostics>Plot Residuals by Normal Quantiles under the red triangle at the top of the output. This plot shows that the normality assumption of the residuals is very reasonable. Since the assumptions of normality and homogeneity of a variance in the residuals are appropriate, we have no concern regarding the statistical tests and confidence interval estimates we’ve obtained earlier. It is important to note, however, that inferences about means, as we’re doing in this ANOVA model, are not very sensitive to the normality assumption. It is still important to check, since serious departures from normality may require alternative analysis techniques. 16 Graphical and numerical description of fitted model When a statistical model is fit to data from an experiment, as was done here, it can be advantageous to create a visualization of that model. Exhibit 14 provides two such visualizations. Exhibit 14 Fitted ANOVA Models To create, save the Predicted Values from the Model by selecting Save Columns>Predicted Values from the top red triangle. Graph>Graph Builder. Place Predicted %Brightness as the Y variable, Temp as the X variable, Time as the group variable, and Agitation as the overlay variable. Select both Points and Line of Fit in the graph palette. To add the data labels, right-click on the variable and choose Label/Unlabel. Next, highlight all the data in the graph, right-click and choose Rows>Row Label. Note: Any combination of choosing the roles for the experimental factors (e.g., X variable, group variable, and overlay variable) can be used. A good practice is to choose a layout that best communicates the features you wish to convey. For example, the second graph has Time as the X variable, which makes it easier to visualize the effect of Time on %Brightness. These visualizations show the effect that each factor has on %Brightness. Notice that the effect of each factor is the same across any levels of the other factors, which is the case since there are no interaction terms in the model. This effect can be verified by calculating the difference between the predicted values 17 for different experimental conditions. For example, the difference between the predicted values between 20 minutes Time and 40 minutes Time is 2.57 for any combination of Temp and Agitation. Another common approach is to display the fitted model with the observed data, as seen in Exhibit 15. Exhibit 15 Observed Response with Fitted ANOVA Model To create, save the Predicted Values from the Model by selecting Save Columns>Predicted Values from the top red triangle. Graph>Graph Builder. Place both %Brightness and Predicted % Brightness as the Y variable, Temp as the X variable, Time as the group variable, and Agitation as the overlay variable. Select both Points and Line of Fit in the graph palette. Points and mean lines will be made for both the Actual %Brightness and Predicted %Brightness values. Right-click on the graph and select Customize. The points for the Predicted values can be removed by choosing transparency of 0 for each Agitation level. Desired colors, markers, and line styles can be chosen. Note: Any combination of experimental factors to be used as the X variable and the two levels of the group variable can be used. A good practice is to choose a layout that best communicates the features you wish to convey, such as assigning the X and overlay variables to the factors you think will be affected most. This graph provides a visualization of how the model is describing the observed data. Comparing this graph to Exhibit 2, where the lines plotted are at the sample means for the 18 different experimental conditions, we can see that our chosen model is essentially a simplified description of data, one that reduces the data to an equation that has only three terms to describe it (main effects for Temp, Time, and Agitation). You may note that for some specific experimental conditions, the fitted model is either entirely below or above the observed data. We had conducted statistical tests to determine if a more complicated model – one with interaction terms – was needed, but those tests were not statistically significant. If we conduct more experimental runs, however, we may detect interaction effects if any truly exist. But at this point, we don’t have statistical evidence to support it, and our main effects model is a very reasonable description of our experimental results. 18 Summary Statistical insights Our analyses provided quite a bit of statistical evidence demonstrating an improvement in the new formulation. The resulting statistical model that we constructed provides an equation that we used to describe that improvement both visually and quantitatively. We learned that this improvement was not the same across the different washing conditions (Temp, Time, and Agitation). For example, the model helped us identify those conditions where the improvement was the most (Cold Temp, 20 minutes Time, and Low Agitation). For this condition, our predicted %Brightness is 21.5 with a 95% CI for the mean response of [20.6, 22.5]. The conditions we predicted to have the lowest improvement was for the Hot Temp, 20 minutes Time, and High Agitation. Here the predicted %Brightness is 10.3 with a 95% CI of [9.4, 11.3]. Based on these results, we can conclude that a consumer would experience an improvement in %Brightness between 9.4% to 22.5% using the new detergent formulation compared to the current one. Implications and further study As in all statistical analyses, it’s important to not only identify statistically significant results but to compare these to what would be practically important. For example, if the objective was to develop a formulation where the improvement in %Brightness was at least 15% across all washing conditions, then we can see that was not achieved. In addition, it is important to remember that the response variable %Brightness is a measure of the percent difference in the Brightness between the two formulations. Thus, we can only describe the difference in Brightness between the two formulations for the different washing conditions. We do not, for example, know from these data and resulting analyses which washing conditions produced the highest or lowest Brightness values for either detergent formulation. To further explore the effectiveness of this new formulation, the R&D team had also conducted this same experiment using four additional stain types (Coffee, Grass, Grease, and Wine). The following exercises will ask you to build a new ANOVA-based statistical model incorporating the different stain types and then use that model to describe the performance of the new detergent. Exercises Use the data in file Cleaning Effectiveness_2.jmp data set to answer the following questions.: These data contain the variables: Stain type (Dirt, Coffee, Grass, Grease, Wine) Water temperature (Cold, Warm, Hot) Washing time (20 minutes, 40 minutes) Level of agitation (Low, Med, High) Replicate of each experimental condition (1, 2) Stain Temp Time Agitation Rep %Brightness The percent increase in brightness of the sub-specimen washed using the new formulation of the detergent versus the current formulation 1. Create a graph that display the %Brightness for each possible treatment combinations. Describe some of the features the graph reveals. 2. Create a table displaying the mean and range %Brightness for each of the possible treatment combinations. 19 3. Build a linear model that describes the impact the experimental factors have on the %Brightness: a) Start by fitting a model that contains the four mains effects (Temp, Time, Agitation, and Stain) as well as the six possible two-way interactions (Temp*Time, Temp*Agitation, Temp*Stain, Time*Agitation, Time*Stain, Agitation*Stain). b) Remove the non-significant two-way interaction terms using a significance level of 0.10. c) Examine the tests for any of the main effects that are not included in any significant interaction terms still in the model. Remove any non-significant terms for these main effects. 4. Evaluate the assumptions of the ANOVA model (i.e., homogeneity of variance and normal distribution of the residuals). 5. Using the Prediction Profiler, determine which Stain is estimated to have the greatest improvement in the new formulation versus the current formulation. For which of the experimental conditions in Temp, Time, and Agitation is that improvement the most Estimate through a 95% confidence interval the average percent increase in %Brightness in the new formulation compared to the current one. 6. Conduct multiple comparisons to determine if there is any statistical difference between the nine different combinations of Temp and Agitation levels for the Stain. 20 SAS
and all other SAS Institute Inc. product or service names are
registered trademarks or trademarks of SAS Institute Inc. in the USA and
other countries. indicates USA registration. Other brand and
product names are trademarks of their respective companies. Copyright
2021, SAS Institute Inc. All rights reserved.