← Back to Archive

What a designed experiment cannot tell you

The limits of the method, stated plainly.

A designed experiment cannot tell you why something happens, only that it does, and it says nothing at all about conditions outside the range of settings it actually tested. No amount of careful design or analysis removes either limit, and both are worth stating plainly, since a study that has earned trust within its own boundaries is too easily assumed to have earned it everywhere.

A hearing test in a soundproof booth

A hearing test in a soundproof booth reliably measures the quietest tone a person can detect at each pitch tested, and the result is trustworthy for exactly those conditions. The test says nothing about why hearing works as it does, which structures in the ear are responsible, or what has changed if a result differs from last year's. Nor does it predict how well the same person will follow a conversation at a busy dinner table, a situation the booth was never built to represent. Both gaps sit in the test's design, so repeating the booth test more carefully closes neither.

Correlation inside a box

A well-run factorial or fractional study produces trustworthy statements about how an outcome changes as its factors are varied across the settings actually tested. It does so without needing to know the physical or chemical reason behind the behaviour, because the method works by observing how settings and outcomes move together. That is a real strength wherever the mechanism is unknown or too complicated to model, since the experiment still delivers a usable, predictive answer.

It is also the boundary. A statement that this setting produces this outcome carries no explanation inside it, and extending it to a different material, a different scale or a setting outside the range tried requires a theory of the mechanism the experiment was never built to supply. Even the gap between two tested settings is only safe to fill in if the underlying relationship is known to behave smoothly across it.

Just past the edge of the tested range

A relationship that looked smooth and straight across the tested settings has no guarantee of staying that way even slightly beyond them, and plenty of real physical relationships bend sharply, level off or reverse just past the edge of where anyone looked. A prediction made just outside a tested boundary is often treated as barely riskier than one made just inside it, although one rests on data and the other on an assumption that nothing unusual waits past the edge.

Used well, a result is applied to predict outcomes inside the tested settings and treated as a starting hypothesis whenever a mechanism needs to be understood or a result extended somewhere new. Pairing it with a separate line of physical reasoning about why the pattern exists is what lets it be extended with some justified confidence.

These limits belong to every empirical method built on observed correlation. A designed experiment's advantage over informal trials is that the tested range is written down before the first run, so the boundary is visible, and a result stretched beyond it without anyone noticing is a failure to respect a line the method took care to draw.

More on Numbers that mislead