This has kind of been asked before in different ways, but I'm struggling to adapt the answers to my specific scenario.
Say I have a ground station and a plane in the sky. I know the GPS lat, lon, and altitude of both. I'm trying to derive the XYZ position of the plane relative to the ground station on a new grid system where the ground station sits at (0,0,0), the Z-axis points out of the ground station into the sky, the Y and X axis are tangent to the altitude of the ground station, and the Y axis projects over the north pole (points north when zoomed in close).
Here's a picture to try and convey this:
The orange dot is the ground station, the brown dot is the plane, and the red axes are how I'd like to define my new relative grid.
I tried doing this: Convert the lat/lon/alt to earth XYZ coordinates, then rotating about the earth Z axis by ground-station longitude, then rotating about the earth Y axis by ground-station latitude, then translating along earth X axis for ground-station altitude (plug earth radius). I then relabeled the axis: earth-X = Z2, earth-Y = X2, and earth-Z = Y2. I got a result that placed the ground station at (0,0,0), but the plane results don't look right: the altitudes shown on the Z-axis are out of order compared to the plane's GPS altitude readings. I'm testing with plane coordinates that are close enough to the ground station that the curvature of the earth should not affect the Z-axis very much (maybe a couple hundred feet, while these planes are at 5000ft to 48000 ft).