Ok so, we have an update. I finally went to bite the bullet and took the car to the Subaru dealership. $85 and 20 minutes later they told me the valve for the turbo needs to be replaced, and that should solve my problem.
Does this sound like a reasonable fix? I'd like to hear the opinions from you guys before I go spending $300 to get it fixed.