The autopilot helps you fly in a straight line, but doesn’t absolve you of your responsibility to look out the window and make sure you’re clear of terrain, obstacles, and other planes. Visual Flight Rules means the pilot in command is responsible for maintaining safety at all times!
You can use it, or not use it, as much as you like to reduce your workload in managing the controls.
Likewise it’s not required to use autopilot for IFR, as long as you monitor your instruments and stick to your assigned path and altitude. Autopilots are used because they make flying safer, easier, and less error-prone.