I have touched/accidently picked up many bristle worms and they aren't so much toxic as irritating (unless you're allergic I suppose) and the chances are pretty good you have lots more then just one if you have live rock in the tank.
If you do brush up against a bristle worm I've found scotch tape works pretty good at getting the bristles out
