I think a custom script might be needed for this which allows you to set fixed layers and use these in your model without having to constantly specify them. You can create one from:
Processing Toolbox > Scripts > Tools > Create new script
And use something like the following:
##Example=name
##Listed_Building=output vector
##Building_Centre_Points=output vector
from qgis.core import QgsVectorLayer
listed_building_path = "path/to/listed_buildings.shp"
centre_points_path = "path/to/centre_points.shp"
Listed_Building = QgsVectorLayer(listed_building_path, 'listed_buildings', 'ogr')
Building_Centre_Points = QgsVectorLayer(centre_points_path, 'centre_points', 'ogr')
Make sure the script is saved in your /.qgis2/processing/scripts/
directory. Then add the script into your model, replacing your input vector parameters.
Example:
If I want to run the Fixed distance buffer tool on two specific layers, I would use the script above, add it into the model, add the buffer algorithm and specify the relevant input layer from the script (e.g. listed buildings):

And repeat for the centre points

Now when I run the model, there's no need to specify the input layers. I just have to assign the output paths:
