The other way round doesn’t feel as strange because all kinds of aircraft; gliders, fighters, transport aircraft and bombers can be equipped with a stick IRL.
Yokes on the other hand are of a very limited use concerning aircraft type.
It doesn’t feel ‘wrong’ to fly e.g. an A300 or A310 with a stick, but flying an Extra or Pitts with a yoke does.