I've never had mine reach "plague" numbers and almost always see a few to a few dozen on the glass. I must have the algae-eating variety, since never found them to pose any danger to my zoas or other corals [knock on wood]
Anything that gets into the 1000s in our tanks would cause a major problem I would imagine. A few bristleworms, good for scavenging uneaten foods. Thousands of them make touching anything in your tank a major hazard. Even certain types of corals. I love pulsing xenia but if they reach into the 1000s, then they are a weed that smother everything else and turns your tank "brown".
As with most things in life, moderation is better than overwhelming amounts.
Anthony