Solution: Geometry Generator for performance
The solution by christophe using SVG is great for static output, let's say for print. However, in this project with several hundred symbols, rendering all the svg symbols including different size and containing attribute data takes enormously long and every change in the canvas (pan, zoom) starts a new rendering cycle. So for dynamic use, Geometry Generator (see here for details) has some advantages. You can use geometry generator to have full controll of all settings.
How to implement it
Text size
Update: Further styling and setting scale visibility allows for fine tunig, see an example here (zoom in to see the text diagrams appear one after the other - first the bigger ones, by zooming more the smaller ones as well): https://qgiscloud.com/daur/textdiagramm_cd/
The solution by christophe using SVG is great for static output, let's say for print. However, in this project with several hundred symbols, rendering all the svg symbols including different size and containing attribute data takes enormely long and every change in the canvas (pan, zoom) starts a new rendering cycle. So for dynamic use, geometry generator has some advantages.