Not as visually cool but at least it work, use the MoveLayerEx in IMapLayers can be used to move a layer in a group layer, use the same grouplayer as source and target and apply the index that the layer shall have in the group. Made a small code sample:
Private Sub MoveGroupLayerItem(ByVal up As Boolean)
Dim layer As ILayer = getLayer()
Dim layerIndex As Integer = getLayerGroupIndex()
Dim map As IMap = axMapControl1.ActiveView.FocusMap
Dim mapLayers As IMapLayers = CType(map, IMapLayers)
Dim groupLayer As IGroupLayer = CType(map.Layer(0), IGroupLayer)
Dim compLayer As ICompositeLayer = CType(groupLayer, ICompositeLayer)
If up Then
If layerIndex > 0 Then
mapLayers.MoveLayerEx(groupLayer, groupLayer, layer, layerIndex - 1)
End If
Else
If (layerIndex < compLayer.Count - 1) Then
mapLayers.MoveLayerEx(groupLayer, groupLayer, layer, layerIndex + 1)
End If
End If
axMapControl1.Refresh(esriViewDrawPhase.esriViewAll, Nothing, Nothing)
End Sub