If you go the route of external 5.1, keep in mind you can use a headset with “open” headphones which are very good at letting external sound in, so you will be able to hear both speaker and headset sound.
But should be no problem with mixing both MSFS and vatsim audio on good quality headphones. The strategy though would be decreasing MSFS volume, not cranking up vatsim. BTW, the MSFS headphone simulation muffles the sound all the time in the cockpit, not just when ATC speaks to you.
Good luck