I work at a detail shop, been there 2+ years. I've seen your problem on ALOT of cars&trucks. Like someone else mentioned, its called raildust, and whether you live next to a railroad track or not it doesn't make a difference - when your car was brought to the Subaru dealer it could've been moved by train, and although when you first got the car you wouldn't see the spots they get noticably worse as the weather gets a shot at your paint.
Clay bar is about the best way to remove, a tip though, instead of using the supplied water your claybar kit came with, use soapy water. I get all my supplies from Carbrite or Carbrite.com since the truck stops by every week. There are a few different grades of claybars and in your case i'd use a pretty aggressive one but be sure to wax the car using an orbital buffer at high RPM afterwards.
If worse comes to worst, and although many many people get scared of the idea, i've used it countless times but regular Automotive Paint Thinner works wonderful on situations where nothing else seems to touch it. If you plan on using this method don't be scared to death but be aware if you dont use it right you can really mess some stuff up.