You’re understanding is correct. You do not need clearance to go under the Class Bravo airspace. What I’m missing though is you are saying when handed off from tower to approach. If this is VFR as you stated, in the real world, tower would simply state “you are leaving my airspace, frequency change approved”. the rest would be up to you.