1. Download all building polygons within the extent of the address point layer.
The QuickOSM plugin allows you to use the extent of a layer in a query.

2. Optional. If the building polygon layer from step 1 is an extremely large layer, it may slow down or even crash QGIS. If this seems like it won't be a problem for your system, skip to step 3.
Buffer the address point layer. For buffer distance, estimate the maximum distance between address points and their corresponding building polygons.
Use the Select by location tool to select building polygons that intersect the buffer.
Save the selected building polygons as a new layer. Use this layer instead of the original building polygons layer.
3. Do a nearest neighbor join between the address points layer and the building layer from step 1.
- Use the NNJoin plugin.
- Input layer: building polygons
- Join layer: address points

At the end of step 3, you have a new building layer. Each building polygon has two new attributes: 1) "join_address_field" = the address of its nearest point; and 2) "distance" = the distance to that point.
4. Use the Select by expression tool to select the building with the lowest distance for each address.
"distance" = minimum( "distance" ,group_by:= "join_address_field" )
If your address is broken up into multiple fields, use concat("join_addressfield1", "join_addressfield2", ...)
. Substitute the name of your address field(s).
5. Save the selection as a new layer. Inspect and correct any errors.
In theory you could correct errors before saving the selection, but in practice it's easy to mess up a selection and lose all the work you put into correcting the selection.