I'm interested in a low-cost or open source solution for creating land cover GIS layers that utilize both spectral and textural extraction algorithms. I have used PCI Geomatica, ENVI, and Feature Analyst VLS in the past; however these solutions are a bit beyond my price range, any software recommendations?
You can use GRASS GIS for this which supports texture extraction and image classification based on a radiometric/segmentation approach. For an idea, check this conference abstract, a planned talk at the Geoinformatics FCE CTU 2011.
If I understand you correctly, you are looking for a supervised classification procedure. Some theoretical background: http://rst.gsfc.nasa.gov/Sect1/Sect1_17.html
This is certainly possible through grass: http://grass.osgeo.org/wiki/Image_classification#Supervised_classification_2
As an alternative you could also look at saga (I'm not saying it is better, I just know it better myself), which also plays nicely with qgis and R. There are some video's demonstrating this on this site: http://www.uni-koblenz-landau.de/landau/fb7/umweltwissenschaften/landscape-ecology/Teaching/geostat (download the datafiles to get the presentations).
In all gis programs, what you will do is define a number of reference points or polygons in one type of land, which are then extrapolated to the rest of the area. Here is an example of a landuse classification:
And in fact if you have drawn your training polygons in any gis program, you can use R to predict. Make an overlay with your grids, and then use any prediction system you like (eg rpart if you want classification trees). More info in this book around page 222: http://www.lulu.com/product/file-download/a-practical-guide-to-geostatistical-mapping/14938111
There is a lot more to say, you training sets should be representative for your study area (perhaps it would even be better to generate random points in R and to classify those). You should also choose your auxiliary datasets carefully, and you might want to generate new ones if eg texture is an important property.
If all you want to do is extract regions or features (without classifying them), a segmentation algorithm is more likely what you want. One example (implemented in SAGA GIS) is discussed in this paper: http://mirror.transact.net.au/pub/sourceforge/s/project/sa/saga-gis/SAGA%20-%20Documentation/GGA115/gga115_03.pdf
You would be able to do that with GRASS.
You will first work with raster data :