General advice: try via the QGIS GUI to achieve what you want then see how to implement it in PyQGIS. It's what I've done here.
You do not need to use any code in particular, just using an expression for fill color style is enough.
Without caring about using PyQGIS first, just try manually the following:
- On your layer you want to select, just set the fill color expression with
- Select features graphically using QGIS GUI
- Switch to your layout (I suppose it contains a map element that display your layer)
- Refresh your layout to see style has changed for selected feature(s)
At PyQGIS level, to change style in expression, you should use a code like below
layer = iface.activeLayer()
# tested for single symbol, single marker here not for more complex styles
propertyColor = QgsProperty()
propertyColor.setExpressionString("if(is_selected(), color_rgb(255,0,0), color_rgb(0,0,255))")