Applies to the current development version or the upcoming release 2.0.
You can use QgsLegendInterface.addLegendLayerAction for this purpose. This method lets you add new menu entries for a LayerType (VectorLayer, RasterLayer, PluginLayer)
a = QAction( u"My Action", iface.legendInterface() )
iface.legendInterface().addLegendLayerAction( a, u"My Plugin Menu", u"id1", QgsMapLayer.VectorLayer, True )
If you set the allLayers parameter of addLegendLayerAction to False you can add the action only for specific layers. Given a variable myLayer pointing to a QgsMapLayer:
aLyr = QAction( u"My Layer Action", iface.legendInterface() )
iface.legendInterface().addLegendLayerAction( aLyr, u"My Plugin Menu", u"id2", QgsMapLayer.VectorLayer, False )
iface.legendInterface().addLegendLayerActionForLayer( aLyr, myLayer )
You would then normally connect the QAction.triggered signal to a slot with the code you want to execute when your menu entry is clicked.
You should remove the action by calling the appropriate removeLegendLayerAction when your plugin is exiting.
iface.legendInterface().removeLegendLayerAction( a )
iface.legendInterface().removeLegendLayerAction( aLyr )