Geoserver allows publishing OGC compliant services like WMS, WFS and WCS. It is open source and has many publishing output types such as OpenLayers. Geoserver can be installed on its own server and then accessed via REST APIs, QGIS and ArcGIS Desktop and Pro. Geoserver can also serve up tiles with an extension. A typical open source tech stack is QGIS client, Postgis DB and geoserver on (Linux).
ArcGIS server is proprietary software that needs to be licensed. An enterprise license could cost somewhere around 10k (half that for dev). In my opinion, it is great software and has grown a lot over the years. It can be installed separately but has more functionality with the other Enterprise components. This GIS server outputs OGC compliant services and proprietary services like map, feature access, geocode, geoprocess and image (vector -- ortho/elevation (i.e. tiles)). ArcGIS is for both raster and vector.
Tileserver is also open source for serving tiles. I don't believe that it serves up other GIS data types like raster.
Overall, you'd have to ask yourself or your company what the business need is for serving up GIS data. In my opinion, ArcGIS server is the most reliable and has the easiest support structure. Geoserver and Tileserver, are cool, but require more technical involvement and the maintenance could be challenging.
One other point, since Geoserver and ArcGIS server use different ports and file systems you could probably install both on the same server to test.