Here’s a little secret - it’s always in windowed mode. The Full Screen option is really a borderless window.
But as others have said, if you really want to run it in normal windowed mode, you can either set it in the menu as demonstrated a couple of posts up, or hit ALT + ENTER to switch back and forth from “fullscreen” to windowed.