You can publicly access many of the 'advanced' (or PAL) labeling engine functions via several classes, though the settings classes do not have sip (Python) bindings yet:
This is the main PAL engine interface that is really part of QgsMapRenderer, so you will want to gain access to it with the following in QGIS's Python console:
iface = qgis.utils.iface
mc = iface.mapCanvas()
mr = mc.mapRenderer()
le = mr.labelingEngine()
You will generally not need to work with the engine via Python. One aspect of accessing the engine directly that might be useful: gain direct access to the labels that have already be rendered. This is how the PAL label manipulation tools in the toolbar work (from master branch: Pin/Unpin, Show/Hide, Move, Rotate, Change):
le.labelsAtPosition(QgsPoint) # returns a label of class QgsLabelPosition
le.labelsWithinRect(QgsRectangle) # returns list of labels
# e.g. le.labelsWithinRect(mc.extent()) would return all visible labels drawn to canvas
see also: QgsLabelPosition
This is where you can set temporary engine-specific options like search method, show candidates, etc. as you can in the Engine settings dialog. There is currently no Python binding for this class.
Having a sip file created (Python binding) for this class, and the following, is the feature request you will want to submit, as per @lynxlynxlynx's suggestion.
This is where you can access, manipulate and write PAL settings for a given vector layer (which I think is what you are looking to do). Unfortunately, there is currently no Python binding for this class, which is defined in the QgsPalLabeling file, and I don't think there is a means of accessing it via another bound class.
Once a binding is in place, you will probably be able to access PAL layer settings like:
palyr = QgsPalLayerSettings()
palyr.readFromLayer(iface.activeLayer()) # assuming active layer is of vector type
if not pallyr.labelPerPart: # access "Label every part of a multi-part feature"
pallyr.labelPerPart = True
# you have just adjusted settings like via the GUI, now redraw the canvas
Check the the QgsPalLabeling class's header and source file comments for info on defaults for QgsPalLayerSettings options.