Works for me:
valLabel=QgsPalLayerSettings()
valLabel.readFromLayer(iface.activeLayer())
valLabel.fontSizeInMapUnits=True // change to map units
valLabel.textFont.setPointSize(100000) // set font size
valLabel.writeToLayer(iface.activeLayer())
iface.mapCanvas().refresh()
For the data-defined way check QgsPalLayerSettings::DataDefinedProperties FontSizeUnit
and enum QgsPalLayerSettings::SizeUnit
Units used for option sizes, before being converted to rendered sizes.
Enumerator
Points
MM
MapUnits
Percent