If they are making comments about either your race or sexuality you could call the police, maybe you can also do that for sexist and abusive comments. I was under the impression (may be an urban myth) that it is now illegal for building site workers to wolf whistle at women from scaffolding etc. I'm not sure the swearing at them would do any good, probably just egg them on, if you feel that you are being targeted unfairly and it is distressing for you, why don't you try the police and make an offical complaint.