The RandomForests algorithm is often used in forestry. There are two implementations of the randomForests algorithm that I regularly use. The first is a pixel-based classifier implimented in R using the randomForests package. I believe this is most sophisticated and flexible approach you are likely to find. There are a many resources to get you started using this route. The following publication used a variety of tree detection algorithms, including randomForests, to identify juniper cover:
Poznanovic, A. J., Falkowski, M. J., Maclean, A. L., Smith, A. M., &
Evans, J. S. (2014). An Accuracy Assessment of Tree Detection
Algorithms in Juniper Woodlands. Photogrammetric Engineering & Remote
Sensing, 80(7), 627-637.
Here are some additional publications that may be of interest:
Immitzer, M., Atzberger, C., & Koukal, T. (2012). Tree species
classification with random forest using very high spatial resolution
8-band WorldView-2 satellite data. Remote Sensing, 4(9), 2661-2693.
Rodriguez-Galiano, V. F., Ghimire, B., Rogan, J., Chica-Olmo, M., &
Rigol-Sanchez, J. P. (2012). An assessment of the effectiveness of a
random forest classifier for land-cover classification. ISPRS Journal
of Photogrammetry and Remote Sensing, 67, 93-104.
The second approach is a object-oriented implementation of the randomForests algorithm in eCognition Developer (There is actually a pixel-based implimentation too). This is a very powerful approach, although is very difficult to implement as there is little documentation. the best advice I can give you is to join the eCognition community and search for "CART, SVM & RF Classifier Example (Improved in eCognition 9.0)" to find the example ruleset. This example with sample imagery included will walk you through how to perform a pixel-based or object-oriented randomForest classifcation.