will filter legend items to those layers which are spatially located i.e. have features which are within the area shown in your layout map, it won't filter out layers which are not checked in your table of contents panel.
setLegendFilterByMapEnabled(True) filters the legend items to show just the ones visible in the associated layout map view. So, if a layer is not checked it will not be visible in the map view and therefore will not appear in the legend. Additionally, if no features of a layer or category thereof (e.g. for layers rendered with categorised/ graduated symbology) are visible in the associated layout map view, the legend node will be filtered out even if it is still checked in the main layers panel.
So, in retrospect, for most applications
setLegendFilterByMapEnabled would probably do exactly what you want. You just have to call
refresh() after setting it enabled:
To add only layers checked in the layers panel to your legend, you need to do something like this:
#get checked layer tree layer objects and store their names in a list
checked_lyrs = [l.name() for l in QgsProject().instance().layerTreeRoot().children() if l.isVisible()]
#get map layer objects of checked layers by matching their names and store those in a list
lyrsToAdd = [l for l in QgsProject().instance().mapLayers().values() if l.name() in checked_lyrs]
legend = QgsLayoutItemLegend(layout)
root = QgsLayerTree()
for lyr in lyrsToAdd:
#add layer objects to the layer tree