I just bought my 04 wrx used with 80k miles. Ive been driving it for a few days and have noticed this slight grinding noise when going downhill or just letting off the gas. I cant hear it over the radio so its not a big deal. But this morning (its like 35 outside) I let the car warm up for about 5-8 minutes but when i was driving it, the grinding noise was incredibly loud and even when i shifted gears it made the sound. But on my drive back from dropping off my sister (only like a 10 minute drive) it went back to its slight grinding noise just when decelerating.
My dad has worked on 2 wrxs the same year and said hes heard the noise is those 2 wrx also. But this morning just scared me a bit til it randomly went back to normal again.
If it helps at all the clutch is new and resurfaced flywheel but it sounds more coming from the center of the car.