I have three layers:

1. polygons (having a unique id)
2. stops (having the polygon id it is laying on) (point)
3. possibilities (having the polygon id it is laying on) (point)

I now want to find the closest possibility-point of each stop-point, which lays on another polygon.

I could easily find the closest point overall, but how could I add the requirement that this point has a different polygon-id than the stop-point itself has?

Here is an example of what I have:

And what I would like to achieve:

I could also use street data from OSM (laying between the buildings) if that would help.

Preferably I would like to be able to integrate this process in a graphical model in QGIS 3.4. But if that's not possible, a different solution than processing tools (like plugins or virtual layers) would do it as well.