I thought I would see if I could replicate this myself, as I had never noticed it, and yes, it was obvious in the stick, and throttle.
I use FSUIPC, so this could make a difference, but I used the A320 ECAM flight control screen to observe the delay as well.
To my surprise, the flight control surface indications had no visible delay, so it seems you may have something there.