Are you sitting right next to your router? Line of sight is recommended, although I have been able to get away with playing some Half Life Alyx in the next room.
I was initially having similar issues - it would seem to work well, low latency high bandwidth etc, but would randomly hiccup. I finally caved in and bought a new router (recommended by VD developer about £40) and had no issues since. Have you tried swapping your 2 routers to see if the other one is any better?
Also you could try the official air link which is free, to rule out any issue with VD itself