I'm not sure if you are programming in ArcMap VBA or Visual Studio?
In VBA you could create a UITool on mouse down event which searches for, I'm assuming a certain layer or project area layer that is selected or not. You could assign the selction count to a function then define your button's enable or disable state according if the function returns 0 or >0.
In Visual Studio same concept applies, however you will use a BaseTool instead of a UITool.
Below is an ArcObjects code sample of how to do a feature selection with a UITool in VBA.
Private Sub UIToolControl1_MouseDown(ByVal button As Long, ByVal shift As Long, ByVal x As Long, ByVal y As Long)
Call layerselection
End Sub
Private Function layerselection() As Integer
'arcmap command for featue selection tool
Dim pUID As New UID
Dim pCmdItem As ICommandItem
' Use the GUID of the Save command
pUID.value = "{78FFF793-98B4-11D1-873B-0000F8751720}"
pUID.SubType = 3
Set pCmdItem = Application.Document.CommandBars.Find(pUID)
pCmdItem.Execute
' Part 1: Set the variables.
Dim pMxDoc As IMxDocument
Dim pMap As IMap
Dim pActiveView As IActiveView
Dim pFeatureLayer As IFeatureLayer
Dim pLayer As ILayer
Dim pEnumLayer As IEnumLayer
Set pMxDoc = ThisDocument
Set pMap = pMxDoc.ActiveView.FocusMap
Set pActiveView = pMap
Set pEnumLayer = pMap.Layers
Set pLayer = pEnumLayer.Next
Do Until pLayer Is Nothing
If pLayer.Name = "MyLayer" Then
Set pFeatureLayer = pLayer
ElseIf pLayer.Name <> "MyLayer" Then
layerselection = 0
Exit Function
End If
Set pLayer = pEnumLayer.Next
Loop
' Part 2: Select features.
Dim pQueryFilter As IQueryFilter
Dim pFeatureSelection As IFeatureSelection
Set pFeatureSelection = pFeatureLayer
pFeatureSelection.SelectFeatures pQueryFilter, esriSelectionResultNew, False
' Refresh again to draw the new selection.
pActiveView.PartialRefresh esriViewGeoSelection, Nothing, Nothing
layerselection = pFeatureSelection.SelectionSet.count
End Function