This is because arcpy.Rename_management does not work on a map layer object - it works on objects on the disk.

Instead try:

    layer.name = newName

`layer.name` is a read/write attribute of the [arcpy.mapping.Layer][1] class.


  [1]: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Layer/00s300000008000000/