Fair enough I suppose.So is stuff like funk, soul or r'n'b and I don't see nearly as many people hating those. I think it's more that most people expect music to have a catchy tune you can wake up stuck with in your head. A lot of rap doesn't really deliver that.
I'd say a similar number of people (likely more) hate metal because "it's just noise" and that genre has fairly few people of colour in it, so I'm not sure we can pin this particular thing on racism, as tempting as it may be.