The most terrible mistake you can make is to consider the geographical coordinates as if they were flat.
To apply transformations on geographic coordinates, spherical trigonometry can be used. I do not know transformation matrices for three dimensions in spherical coordinates. Still, I could develop the formulas to establish a rotation of one point on another a certain angle. But I'm sure that's not what you need.
I recommend, on the other hand, to transform the geographical coordinates into flat ones through a projection. Perform a planimetric rotation on said projection, and then anti-project the results towards geographic coordinates again.
What is said, x is not equal to longitude; y is not equal to latitude.
y_1 the planimetric coordinates of a point. You can calculate the rotation of an angle
theta around the origin of coordinates. Consider
y_~1 the coordinates resulting from that transformation. Therefore:
x_~1 = x_1 * cos(theta) - y_1 * sin(theta) y_~1 = x_1 * sin(theta) + y_1 * cos(theta)
If you want to rotate around a point whose coordinates
y_0 do not correspond to the origin of coordinates, then you must compose three transformations: translate the center of rotation to the origin of coordinates, rotate on it, translate back to the initial position.
x_~1 = ( x_1 - x_0) * cos(theta) - (y_1 - y_0) * sin(theta) + x_0 y_~1 = ( x_1 - x_0) * sin(theta) + (y_1 - y_0) * cos(theta) + y_0
Finally, if you want to rotate around the centroid of the polygon, but you do not know its coordinates, you can calculate them as the algebraic sum of the coordinates of each vertex, divided by the number of vertices.
x_0 = ( x_1 + x_2 + ... + x_n ) / n y_0 = ( y_1 + y_2 + ... + y_n ) / n