Has it done this for the entire time you have owned the car or did it start doing it recently? If so, what changed? Did you do something to the car and then it started doing it or did it just appear out of nowhere? A little more info would be appreciated...kinda sounds like just a driver thing. Maybe ask someone else to drive your car and see if they notice it. Doesn't sound serious though.
Oh, and if you have any mods or anything that isn't totally stock please post up so we can try to include that in a diagnosis.