Are you connected to a server that has the quickest response time?
If you ever manually selected which server you connect to (I once did when East US server wasn’t responding well), you then attempt to connect to that server each time. Automatic will select the best server at the time. So after the one time I manually selected a server, I had to go and select back Automatic to keep it from trying to reconnect to that server.
Since you have good bandwidth, yet your download times are atrocious, I am thinking making that check might help.