I actually wrote an application that did just this a number of years ago. It was to label oil and gas wells with various attributes. It was way back in Avenue days and was proprietary in any case.
But the way it worked was to get the extent of the label as a polygon and spiral around the point looking for overlapping labels. It is a matter of testing the location in a spiral pattern. The label would get placed in the first location where there weren't any overlapping labels, using a callout or not based on how far the label was from the point. You also have to note the angle from the point so you know where to attach the callout. The spiral kept moving outward from the point until it found an empty place to put the label. I imagine I had a ceiling of some sort, but I don't remember that being a problem.
Anyway, maybe this information will get you started. In Avenue, there was a labeling machine that I could use as a helper rather than having to program all the geometry from scratch. Maybe ArcObjects has something similar.