As others have said, you're not alone in this. Around here the WRX and STi are the target cars. Muscle cars are owned by *******s and old people in my area, nobody really cares or pays attention to them, but EVERYONE has their eye on the Subaru.
At least twice a week I get the typical asshat who just won't leave me alone. Just last week I had a Nissan SE-R and a Honda Accord trying to hassle me. The odd thing about it is I work nights, so I only see people either late in the evening, 0100-0200 (lunch), or early in the morning rush hour. Typically I get messed with in the 0100-0200 range. It can be very annoying, but it is also pretty easy to ignore. And honestly, I think ignoring them pisses them off more than losing any race would.
Just mind your own business, and do your thing. If they're driving unsafe around you, get off the road for a little bit, and then resume. You're never in too much of a hurry to be safe, especially when your family is involved.