I finally found the issue. Something is wrong with the Seagull model. Just after it loads, the FPS drops.
As a test, I just deleted the model from the game files. (asobo-simobjects-animals\SimObjects\Animals\Seagull\ directory)
And FPS drop is gone!
I have no idea what is wrong with the Seagull model, or even if it might be all the animals, but THAT is the problem.
PS: Should there even be Seagulls even 150nm inland???
Edit: I just deleted all the animals for now. Tested my homebase of FACT where I used to get under 30FPS, now getting 40FPS. It is next to the sea so I assume there will be seagulls. I am actually seeing myself hit other limits now, which is good 
Edit2: Posted as bug @ Seagull model causes massive FPS drop when loaded - #2 by CptLucky8