Active Effects: Not Everything Matters Equally
In most experiments, only a handful of factors actually drive the response and the rest is just noise. This observation is one of the most important principles in experimental design, and it’s the reason why fractional factorial designs and screening experiments work at all.
What is an active effect?
An active effect is one whose effect size is large enough to be practically significant. It makes a real, meaningful difference to your response. An inactive effect might technically exist, but the shift is so small that it doesn’t matter in practice.
The distinction between practically significant and statistically significant matters more than most people realize. Statistical significance tells you whether an effect is distinguishable from random noise (p-value below 0.05). But that doesn’t mean it matters.
Here’s an example. You run a precise experiment with many replicates and find that increasing temperature by 30°C increases your filtration rate by 0.3 L/min. The p-value is 0.001. But your process runs at 70 L/min and you need improvements of at least 5 L/min to justify a change. That 0.3 L/min effect is statistically real but practically meaningless.
The reverse also happens. A large effect of 8 L/min might have a p-value of 0.12 because you only ran a few experiments with noisy data. Statistically not significant. But practically? That’s an effect worth investigating further.
Rule of thumb: Statistical significance tells you whether an effect is real. Practical significance tells you whether it matters. You need both, but practical significance is what drives your decisions.
How do you decide if an effect is active?
There are three main approaches, and in practice you’ll often combine them.
Option 1: Use your expert knowledge
You know your process. If changing a factor leads to a predicted improvement of 0.5 L/min, ask yourself: does that matter? Is it worth the effort of controlling that variable more tightly? This is a judgment call, not a statistical one. And it’s perfectly valid.
Option 2: Compare effect sizes: the Pareto chart
When you calculate all your effects, you can compare their magnitudes directly. A Pareto chart ranks effects from largest to smallest, so active effects are the ones that stand out.
Let’s look at a real example. We ran a full factorial design on a filtration process with four factors: Temperature (T), Stirring Rate (RPM), Concentration of Flocculant (CoF), and Pressure (P). Here are the calculated effects plotted in a Pareto chart.
There’s a clear gap. Temperature, the T × CoF interaction, T × RPM, RPM, and CoF have large effects. Pressure and the remaining interactions are small and likely inactive.
Option 3: Use a half-normal plot
A half-normal plot is especially useful when you have many effects. Here’s how it works:
- Take the absolute values of all estimated effects
- Sort them from smallest to largest
- Plot them against the quantiles of a half-normal distribution
If all effects were inactive (just random noise), they would fall roughly on a straight line through the origin. Active effects deviate from that line and appear in the upper right corner.
The five smallest effects sit neatly on the reference line. Then CoF breaks away, followed by RPM, T × RPM, T × CoF, and finally T at the top. The separation is clear.
The nice thing about this approach is that you don’t need replicates and you don’t need to calculate p-values. The inactive effects serve as their own estimate of noise.
Deep Dive: Why “half-normal”? We use absolute values because we don’t care about the direction of the effect, only its magnitude. The half-normal distribution is the positive half of a normal distribution, which is what you’d expect if all effects were random noise centered at zero.
Effect sparsity
The observation that only a few effects tend to be active has a name: effect sparsity.
Effect sparsity states that in most experiments, only a small proportion of factors have significant effects. This has been confirmed across engineering, chemistry, manufacturing, and many other fields. The more factors you test, the more pronounced this becomes. If you’re testing 15 factors, three to five of them will typically drive the response. The rest are noise.
Effect hierarchy
There’s a related principle called effect hierarchy. It states that lower-order effects tend to be larger than higher-order effects. In other words: Main effects are usually larger than two-way interactions, and two-way interactions are usually larger than three-way interactions. By the time you get to four-way or five-way interactions, they’re almost always negligible.
Rule of thumb: Main effects explain the majority of variation in most experiments. Two-way interactions explain most of what’s left. Three-way and higher interactions are rarely important.
Why this matters for experimental design
These two principles are the foundation of efficient experimental design.
Fractional factorial designs confound higher-order interactions with main effects. This works because effect hierarchy tells us those higher-order interactions are probably negligible.
Screening designs identify active factors using very few runs. They work because of effect sparsity: most factors won’t matter, and you just need to find the ones that do.
Without these principles, you’d need full factorial designs for every problem. With them, you can design experiments that are both economical and informative.
Key Takeaways
An active effect is one that has a practically meaningful impact on your response. Not every statistically significant effect is active, and not every active effect will be statistically significant in a small experiment.
To identify active effects, combine your process knowledge with visual tools like Pareto charts and half-normal plots. Don’t rely on p-values alone.
Effect sparsity (few factors matter) and effect hierarchy (main effects matter most) are the empirical patterns that make efficient experimental design possible.