This issue seems to have been solved already some time ago, but I´ll add my spoon to the soup anyway since I recently needed some raster stats myself.
The QGIS methodology described by rudivonstaden worked like a charm, thanks for the tip. However, in my case I also wanted to know min&max values and std for the raster values within my polygons, and that functionality I could´nt find in QGIS. I found a solution through the function isectpolyrst in GME, which is a favorite of mine for solving GIS problems with.This function gave me just the stats I needed and rather quickly as well.
PS. GME has a bit of an unfortunate dependency (ArcGIS). So even though the program itself is freely available to everyone, you need ArcGIS to run it. But if you do have ArcGIS (any level of licence), I recommend you try it out.
EDIT:
I got the stats I wanted, but I later found out that I jumped to conclusions a little hastily after looking at the actual numbers. You get stats, yes, but only for the first band in a raster. So if you operate with a 3-band raster image, the stats are incomplete. I should have pointed out the number of channels in my original post. Best to use SAGA-GIS like stn proposed above if your working with multi band raster images.