This will depend on your GPU.
For certain there are “sweet spot” drivers for older gpu’s like 1080 which is probably where much of the talk originates.
I swapped out my 1080ti in March(ish) and tested a few drivers across several titles and benchmarks and there were negative performance differences with drivers beyond a certain release date.
I have a utility PC running a GTX 770 and it will actually blackscreen unless it has drivers within a specific range.
With the 3090 in my flightsim rig at present I have no perceivable issues and have whatever drivers were the most current when I turned off the PC last night but it does stand to reason that any patch to fix a microcode exploit issue might also impact performance as this was seen with patches for the same type of exploit with Intel CPU’s.