Try to use the A32NX Custom FBW version instead. If you use the latest installer, it’s the third option. They changed the aircraft fly-by-wire system behaviours, making it fly much stable and more predictable.
I still need to use the spoilers but only for descending below 10,000 ft. Just follow the ATC instructions. Once you hit 10,000 ft and the aircraft slows down to 250 knots. Deploy full spoilers. (I know that in the air, the spoilers only deploy 50%, but that’s not what I’m seeing from the speed brake lever inside the cockpit. I can deploy full in the lever. It’s probably translated to 50% in the speedbrakes itself)
Then when you reach the decel point, keep the speedbrake open as well, to slow down faster and descent faster as well. You just need a bit of practice. Eventually, you’ll get used to it and be able to descent on time.