MapBox open-source stack seems to be quite performant indeed - and used in production by many.
If you want to host your own OSM tileserver have a look at our project http://osm2vectortiles.org/. You can get there downloadable vector tiles for world or countries - and free software to host it (or generate the vector tiles on your own). We work now on implementing updating mechanism - to keep the vector tiles updated regularly.
You have to decide what your tileserver should do:
1) Serve raw vector tiles (/z/x/y.pbf) to native mobile applications (Android/iOS) and websites using the MapBox GL JS (client side rendering with WebGL). Such maps use the JSON GL styles. Raw vector tiles can be also displayed with OpenLayers or Leaflet without WebGL in older webbrowser - but with limited styling capabilities and performance.
Hosting in this case is possible with quite a basic script, even a PHP webhosting could be used: https://github.com/klokantech/tileserver-php or static .pbf files after unpacking the tiles into a folder. For alternatives see: Self-hosting Mapbox vector tiles
2) Serve raster tiles with JSON GL styles. If you want to use on modern clients the above approach, on older clients you can still get the good old raster tiles (/z/x/y.png) openable on any existing mapping library. To create raster tiles from MapBox JSON GL styles on your server you may need to install: https://github.com/klokantech/tileserver-gl
This will serve both the raw vector tiles as well as raster tiles made dynamically on the server with the same JSON style.
3) Serve raster tiles with Mapnik styles. Map style can be written in CartoCSS - typically in MapBox Studio Classic while styling the vector tiles on server. The styling capabilities are more powerful then with the new JSON GL styles - and the raster tiles are always produced on server side. There are many styles with .tm2, which can be used this way. This approach is quite stable, but will probably not be developed very actively further. To check how to host such tileserver easily see: https://github.com/klokantech/tileserver-mapnik/ or documentation in osm2vectortiles project.