This question has been converted to Community Wiki and wiki locked because it is an example of a question that seeks a list of answers and appears to be popular enough to protect it from closure. It should be treated as a special case and should not be viewed as the type of question that is encouraged on this, or any Stack Exchange site, but if you wish to contribute more content to it then feel free to do so by editing this answer.
In a nutshell: Start with QGIS.
There are several Free and Open Source tools for geospatial statistics.
QGIS (Quantum GIS)
There are several spatial statistics plugins in Quantum GIS, such as
- fTools: Tools for vector data analysis and management
- Zonal Statistics: Extended zonal statistics and report generation
- manageR: Interface to the R statistical programming language
- Landscape Ecology Statistics: Contains several analytical functions for land cover analysis
- Live Statistics: display simple statistics about vector data in small toolbars that provide real-time feedback.
- Statist: Calculate and show statistics for a field
Linfinity has provided a brief tutorial covering QGIS Spatial Stastics.
Anita Graser has written a tutorial on the QGIS Group Statistics plugin.
GGIS Processing (model builder)##
QGIS 2.0 Processing is a graphical modelling environment that integrates with several prominent projects, including SAGA and GRASS, some of which contain various statistical algorithms.
System for Automated Geoscientific Analysis
SAGA has over 400 modules for geoprocessing and statistical analysis.
GRASS
GRASS contains several zonal statistics functions.
R Language
The R Language has several spatial extensions.
Spatstat
The Spatstat module for R provides a host of spatial statistics functions.
R Studio
R Studio is a very nice IDE for the R Language, and will help you to easily locate and install the R spatial libraries.