Every experiment is a point in the design space

If you only change one factor at a time, you always know what caused the change in the response. This is what we’ve been taught. It makes sense and is very intuitive. But try to take a step back and ask yourself what an experiment actually is.

The design space

An experiment is one point in a design space. A design space is defined by the factors you choose to vary and the ranges you set for each. For example you might decide to vary curing temperature and curing time in an experiment. Those are the factors. You decide that it makes sense to vary the curing temperature between 60 and 80 °C and curing time between 30 and 60 minutes. Those are the ranges you set for each factor.

In this case, the design space is a square. One side is the curing temperature, from 60 to 80 °C. The other side is the curing time, from 30 to 60 minutes. Every point inside the square is a possible experiment. A combination of the two factor levels. For example a curing temperature of 60 °C combined with a curing time of 30 minutes is one such point.

The design space for two factors. Curing temperature runs from 60 to 80 °C along the bottom, curing time from 30 to 60 minutes up the side, and every point inside the square is one combination of the two factor levels. A single experiment, like 70 °C and 45 minutes, is one point in that space.

One factor at a time covers only the edges

Think about what changing one factor at a time does inside that square. You hold the curing time fixed and vary only the temperature, which moves you along the bottom edge. Then you keep the temperature that worked best and vary the curing time there, which moves you up the right side. You have traced two edges, and nothing else. The whole middle of the square stays empty.

One factor at a time moves along two edges of the design space. You vary the temperature at the low curing time, then keep the best temperature and vary the curing time there, so the runs line up along the bottom and the right side. The whole interior stays empty, and so does the corner at the low temperature and the long curing time.

Four corners cover the whole space

Now think about how you could arrange these design points more effectively. You still only have two factors, but instead of tracing edges, you could place the four experiments in the corners of the square. That way you would cover the entire design space.

A factorial design places one run at each corner of the design space. The same handful of runs now covers the whole square.

This is called a factorial design. Every factor is set to either the low or the high end of the range you chose, pairing the low and high temperature with the short and long curing time.

What one factor at a time measures

The result we care about here is film hardness, measured in pendulum oscillations, where more oscillations means a harder film.

If you run the study one factor at a time, you change one factor, keep the setting that worked best, and carry on from there. Starting in the lower-left corner of the design space, you first increase the temperature while holding the curing time at 30 minutes, measuring the hardness at 60, 70 and 80 °C.

Holding the curing time at 30 minutes, the hardness climbs steeply with temperature, from 48 at 60 °C to 77 at 80 °C.

Because 80 °C gave the hardest film, you hold the temperature there and now vary the curing time, measuring at 30, 45 and 60 minutes.

Holding the temperature at 80 °C, on the same hardness scale, the curing time barely changes the result, from 77 at 30 minutes to 80 at 60 minutes.

One factor at a time stops here. From the two experiments you would conclude that temperature increases the hardness and that curing time doesn’t do much, and you would settle on 80 °C and 30 minutes. One corner of the square is still untested, the low temperature with the long curing time.

What the fourth corner adds

The factorial design spends its fourth run on the upper-left corner, at 60 °C and 60 minutes. With it, you can compare the effect of the curing time at both temperatures. At 80 °C the curing time barely changes the hardness of the film. It only rises from 77 to 80, which is most likely just noise. However, at 60 °C the same change in curing time from 30 to 60 minutes lifts the hardness from 48 to 77.

This also makes sense from a practical point of view. At 80 °C the film cures almost all the way whatever the time, so holding it longer adds little. At 60 °C the film is far from finished at 30 minutes, so the extra time does help.

Hardness against curing time at each temperature. At 80 °C the line is nearly flat, so the curing time hardly matters. At 60 °C it climbs from 48 to 77. The curing time matters only at the low temperature, and the two routes meet at the same 77. The orange point, 60 °C and 60 minutes, is the one run one factor at a time never made.

So the curing time does matter, just not at 80 °C. Its effect depends on the temperature. This is called an interaction.

More information from fewer runs

With this example we now have a choice. We can either run the process at a higher temperature for 30 minutes, or at a lower temperature for a longer time, and reach the same result either way.

We could only see this because we ran a factorial design. One factor at a time, the approach we were all taught and know all too well, would never have shown it.

It also takes fewer runs. The in-between runs in the one-factor-at-a-time study added no extra information. The corners carry the information, and a factorial spends its runs there. You cover the whole space and learn more from fewer experiments.