This is certainly possible with rasters.
This screenshot hopefully shows the issue more clearly. The portion B of the voronoi is closer 'as the crow flies' to the original voronoi centre, but this doesn't take into account the fact that it would take longer to walk around the building. My understanding of the OP's question is that the voronoi needs to take into account this extra distance to walk around the building.

I like the suggestion from @Guillaume. However, when I tried it I had problems getting r.grow.distance
to honour the mask (see below. The ripples shouldn't pass through the buildings).
My Grass knowledge isn't as strong as it could be, so maybe I'm doing something stupid. Definitely, check out that suggestion first as it'll be a lot less work than mine ;-)

Step 1 - Create a cost surface
First step is to create a cost surface. This only needs to be done once.
- create an editable layer, holes and all.
- add a field called 'unit', set it to 1.
- using polygon-to-raster on your "punched out" vector layer (the one which has the holes), using field 'unit'. You now have a layer "mask", where 1 is free space and 0 is building.
use raster calculator to turn this into a cost surface. I'll set 'outdoors' to 1 and 'indoors' to 9999. This will make moving through buildings prohibitively difficult.
(("mask@1"=1)*1)+(("mask@1"=0)*9999)
You can get more 'organic' results by adding a bit of noise to the cost surface (e.g. use random number from 1 to 3, rather than just 1 for outdoor pxiels.)
Step 2. Create cumulative cost rasters for each voronoi center
Now we can run (for one voronoi cell at a time) the GRASS algorithm r.cost.coordinates
against our cost surface layer.
For the start coordinate, use the vornoi center. For the end coordinate, choose one of the corners of your area. I suggest using 'Knights Tour' as this gives smoother results.
The result shows lines of equal travel time from one voronoi center. Note how the bands wrap around the buildings.

Not sure how best to automate this. Maybe processing batch mode, or done in pyqgis.
Step 3. Merge down the rasters
This will probably need code. The algorithm would be
create a raster 'A' to match the size of your cumulative cost images
fill raster 'A' with a suitably high number e.g. 9999
create an array of the same size as the raster.
for each cumulative cost raster number 1..N
for each cell in image
if cell < value in raster 'A'
set value in raster 'A' to cell value
set corresponding cell in array to cum. cost image number
write out array as a raster
That approach should yield a raster where each cell is categorised by the voronoi center it's closest to, taking into account obstacles.
You could then use raster-to-polygon. You could then use the Generalise plugin to remove the "step" effect artifacts from the raster.
Apologies for vagueness on steps 2 and 3... I'm hoping someone chimes in with a more elegant solution :)