Couldn’t it be a modeling issue? Some of the faces could have their normals pointing inside and it would make the normal map and its shading look inverted.
Maybe try this in Blender:
Edit mode, select all faces of your model, recalculate normals outside (shift N).