The center of your head must be at the same location where your camera viewpoint is. Otherwise you will see a distorted picture on the screens from where you sit.
This is the law of optics.
Edit: The camera viewpoint is always perpendicular to the center of the main (middle) screen. The distance from the screen is dependent on the zoom level (field of view).