I'd say that height matters the most in Standard. And even there, it's hip height and leg length that matters, not overall height. This is due to the continous closed contact between partners.
In Smooth, which from a contact point of view is half way between Standard and Latin/Rhythm, it is a little easier to work around height differences.
In Latin/Rhythm there's a lot more leeway for height differences, but even then too much of a difference can make a couple look unbalanced. However, great dancing always powers exceptions to this rule: Look at Andrei Graveline & Elena (whatever her last name is). They're a Russian couple that lives and dances in the US. They won Rising Star Latin at Blackpool not long ago and he's a LOT taller than her. Ditto for the Cutlers from England; Nicole was a lot shorter than the guy (sorry, forgot his given name).