What makes a flag flap
Instability in a flexible sheet, and why the motion grows instead of settling.
A flag flaps because once wind passing over its surface reaches a certain speed, any small ripple in the fabric starts feeding energy back into itself faster than the flag's own stiffness and drag can absorb it, so instead of settling back flat the ripple grows into a full, self-sustaining wave that travels the length of the cloth for as long as the wind keeps blowing.
How a ripple feeds itself
A flag hanging in still air is flat, and in a gentle enough breeze it can stay nearly flat too, with any tiny ripple quickly damped out by the cloth's own stiffness and the drag of the air moving smoothly across it. As the wind rises, a ripple starts to matter more. The part of the ripple facing slightly into the wind is pushed harder than the part facing slightly away, and that uneven pressure pushes the fabric further out of flat instead of back towards it.
Past a certain wind speed the feedback becomes self-reinforcing. A slightly larger ripple makes a slightly larger pressure imbalance, which bends the fabric further, which makes the imbalance larger again, and the stiffness and drag that kept the flag flat at lower speeds can no longer break the cycle. Stiffness stays the same however hard the wind blows, while the push from an uneven airflow grows with every increase in wind speed, so somewhere along the way the push was bound to overtake it. The ripple grows into a wave that runs down the flag's length. Because the wave keeps moving, the pattern of pressure on the fabric keeps shifting with it, so the cloth never settles into a steady curved shape the way a stiff sheet under a fixed load would.
All the energy comes from the wind. A flag flaps harder as the breeze strengthens and goes still within moments of the wind dropping, since it has no store of its own energy to keep the wave travelling.
A sheet of paper in front of a fan
Holding a loose sheet of paper by one edge in front of an electric fan on its lowest setting usually leaves it with only a faint tremble, any ripple near the free edge fading almost as soon as it appears. Turning the fan up changes things suddenly. Past a certain point, a small ripple near the free edge stops fading and within a second or two grows into a large, sustained flap that carries on even with the fan held perfectly steady. Only the air speed changed, and that was enough to flip the sheet from a stable, nearly flat state into a self-sustaining flap. The fan offers a version of the effect that can be switched on and off at will and watched up close, where an outdoor flag is left to whatever the wind happens to be doing on a given day.
That abruptness is the signature of a feedback-driven instability. The flapping does not rise little by little in proportion to the wind, like a dial being turned up. There is a threshold, with stillness below it and flapping above it, which is why a flag can hang limp for an hour on a calm day and then start flapping within seconds of a gust arriving.
A sheet of card held in the same airstream resists far more stubbornly, needing a much stronger blast before any flapping starts, because its stiffness does more of the work of damping out the growing ripple. A flag's fabric weight and stiffness shift its critical wind speed in the same way.
Taut sails and slack ones
A flag cut from very stiff material, or one held taut on a rigid frame instead of hanging free from one edge, resists flapping even in a strong wind, since stiffness and tension both damp out a growing ripple before it can feed back on itself. A correctly trimmed sail holds a smooth, steady shape for the same reason, in a wind that would set a loosely hung flag of similar fabric snapping violently.
Let that sail go slack, losing even a little of its tension, and it becomes open to the same feedback loop, luffing and flapping noisily. Sailors therefore treat an unexplained flapping sail as a tension problem to correct at once, and not simply as a stronger gust passing through.