What can go wrong and what kind of error can build up if I use geodetic coordinates as if they were cartesian and place my objects into my world with no conversion formulas?
Everything really. Distances will become meaningless. Areas will become meaningless. Hence all outputs such as slope will become problematic. As you move in latitude then your "grids" will appear to be the same size but actually one will become a small fraction of the other in true size. I would say distance, area, and angle are the primary errors that will magnify if you do this.
A potential positive is an invite to this prestigious society.
Latitude and longitude are not linear measures, and a planar calculation of distance using them will yield nonsense results. Although lines of latitude, or parallels, are evenly spaced, and could be converted to a linear distance, the distance between lines of longitude, or meridians, varies with latitude. If we apply the formula for Euclidean distance between two points expressed in latitudes and longitudes, we end up with a values that makes very little sense. The Euclidean distance only works for planar coordinates