Why One-Factor-at-a-Time Experiments Fail
Changing one factor at a time seems careful, but it hides interactions and wastes runs. A worked example shows what it misses and what a factorial design finds.
By Covariance Labs · · 2 min read
This guide has not yet been reviewed by an independent statistician. If you spot an error, please tell us.
The usual way to improve a process is to change one setting, see if things get better, keep the change if so, and move on to the next setting. This is called one-factor-at-a-time (OFAT). It is intuitive, and it has two serious problems.
Problem 1: it cannot see interactions
An interaction exists when the effect of one factor depends on the level of another. Here is a small example with two factors, temperature and pressure, each at a low and a high level. The numbers are the yield (%) from four runs:
| Low pressure | High pressure | |
|---|---|---|
| Low temperature | 60 | 65 |
| High temperature | 70 | 55 |
Suppose you start at low temperature and low pressure (yield 60) and follow the OFAT recipe.
- Raise temperature, keeping pressure low: yield goes to 70. A gain of 10, so keep it.
- Now raise pressure, keeping temperature high: yield falls to 55. A loss of 15, so reject it.
You settle at 70% and conclude that high temperature is good and pressure does not help. But look at the whole table. At high pressure, raising the temperature reduces yield from 65 to 55. The effect of temperature flips sign depending on pressure.
Averaged over both pressures, the main effect of temperature is exactly zero: (10 + (−10)) / 2 = 0. A summary of "temperature effect" would be misleading either way. The truth is in the interaction: (55 − 65 − (70 − 60)) / 2 = −10 yield points.
Problem 2: it wastes runs
In a two-level factorial, every run contributes to the estimate of every effect. With four runs the factorial above estimated both main effects and the interaction. Each main effect is the difference between the average of the high runs and the average of the low runs, so it uses all the data rather than just two runs.
OFAT spends its runs on comparisons of two points at a time, and its estimates are noisier for the same number of runs. As factors are added the gap widens: a full factorial in 5 factors takes 32 runs and estimates 5 main effects and 10 two-factor interactions, while a fractional factorial can screen the same 5 factors in 16 runs.
What to do instead
- Use a full factorial for a small number of factors.
- Use a fractional factorial to screen many factors when you expect only a few to matter.
- Use a response-surface design when you are close to the optimum and want to model curvature (see central composite vs Box-Behnken).
You can generate any of these with the free design generator, or read the full overview in our practical guide to DoE.
Frequently asked questions
- What is one-factor-at-a-time (OFAT) experimentation?
- OFAT means changing a single factor while holding all others fixed, then repeating for each factor in turn. It is simple to run but cannot detect interactions between factors.
- Is OFAT ever acceptable?
- It can work when you are confident the factors do not interact, or for a quick feasibility check on a single factor. Most real processes have at least some interactions, which is why designed experiments are the safer default.