I may have confused my second experience with the A300, the cockpits are similar.
I can replicate the issue in the A310, the problem seems to follow the pack valves. They are turn on by default but fed by the APU Bleed. If packs 1 an 2 are turned off the fans turn on but the opposite is supposed to happen.