I'm trying to draw woodlands in OpenStreetMap, using Yahoo sattelite images.
JOSM editor has some plugins that try to automate the process - you have to click inside the area and the plugin finds the boundries. But the quality is pretty bad.
I'm looking for some libraries/algorithms to get good quality boundaries.
The images i'm working with look like this: http://maps.yahoo.com/#mvt=s&lat=56.907056&lon=24.597595&zoom=14
You are better off using a remote sensing application. Of course, you need to have the raster imagery on your computer. There are tons of methods that can help you determine woodland areas, such as: using Neural Networks, trained patches of imagery, supervised/unsupervised segmentation and classification. I'm not sure if this solves your problem, but it's a start.
There are free DIP (digital image processing), such as GRASS, SPRING (I think it's only available in pt-BR) and OSSIM (I'm not sure about this one)
Perhaps try different source imagery. With OnEarth you can pick and choose among different band combinations. The pseudo or false colour ones highlight the differences between vegetated and non-vegetated areas better than the "natural" or "visual" colour combo (scroll down to WMS Global Mosaic use examples). The OnEarth data is available via TiledWMS, KML, and direct download (plain WMS is available too but discouraged to alleviate server load). The imagery is free & libre so there's no worries about what you're allowed to do with it.
To avoid the licensing pitfalls, you can grab plenty of Landsat TM5/ETM7 data from GLOVIS. Then, using eg bands 3 and 4 (red and near-infrarred), and possibly others, you can try to classify the image, export as a polygon and then tweak the polygon to your hearts' content. For forests, using the spatial correlation between pixels is often very useful (in your example, look at the granularity of forest stands). Texture classifiers (for example, calculate the variance of NDVI over a 3x3 window) supplement the pure radiometric classifiers.
Regarding tools, GRASS has been mentioned as is probably a good choice. We have ENVI at work, and while not free software, it would be the tool I'd consider for this.
Note that the Landsat data are often contaminated by clouds or cloud shadow. You may need to dig a bit into the archive to find suitable data.
NASA recently created a global forest height map, perhaps using this as a basis for editing would get you quite far down the road toward your goal.
To derive borders, you are looking for a region growing algorithm. This paper discusses such algorithms, one of which is implemented in SAGA GIS
Like mentioned in other answers, you should indeed try to use more bands than just the visible light. Especially near-infrared and infrared should work well.
And in fact, most gis/remote sensing programs go further: once you have a few example polygons, they can perform 'supervised' classification, which will even suggest new forests. You will find many algorithms if you perform a search on that.
