Ok. I would ensure then that you have things setup correctly before you take the runway for departure. In order for the 320 to properly control thrust and hence speed, you need the Auto Thrust system working. What version of FBW are you using ?
Another thought, are you setting up the MCDU with the required information on the INIT A, INIT B, and PERF pages? Does your FMA and FCU look like this on takeoff? If not, that is your issue with A/THR
