Just try it and see.
You are not mixing different ram in the same bank, you are mixing two different matched pairs in different banks. It might fail to boot. or it might work fine with only a barely noticeable performance hit. If it does boot OK your RAM being 5% or so slower will probably translate to something like a 0.5% or less frame rate hit in game. You will never know till you try it.
The exception will be if you intend to do some serious overclocking in which case you want as many things in your favor as possible including matched ram.