The business no doubt does some harm. But like Ian says it also does some good. Compared to fishing and farm/industrial runoff it is a tiny drop in the bucket. I have swam on reefs that were completely choked in algae. Excess nutrients combined with overfishing of large herbivores is bad news. By-catch from trawling is also really nasty especially the shrimping business.
Many of the fish caught for the trade are very fast breeders and I suspect will fill a territory in a hurry. Some better than others of course. Here is an interesting site to look at, compare these two fish on fishbase. Scroll down and pay attention to the ratings for resilience and vulnerability.
http://www.fishbase.gr/summary/6033
http://www.fishbase.org/summary/1257