Solution with QGIS:
You can use the Qneat3 plugin or native network analysis of QGIS 3 for this purpose. The requirement is that you need to have a road network in order to be able to calculate the distances.
First convert the polygons to centroids or the points that represent the origin and destination. Then use those layers and one of the aforementioned plugins to calculate the distance. If you do not need the shape of the road, then use OD matrix. Otherwise, there are no fast solutions in QGIS.¨
Solution in python:
Use OSMNX library to calculate the distances between points:
you need to use the closest facility analysis of network analysis extension (If you have access to it). Convert the polygon to points, then load them as facilities in the network analysis tool. Load buildings as incidents and then run the analysis. This can provide you with either shape of the paths or just the distances in values.
I personally suggest Python as it is much faster and you have more control over it, but it is more complex. QGIS is free and fast but may not offer many options. ArcGIS is a bit slower compared to others, but Interface is easy to work with and offers many options, such as finding several facilities etc.