For the raster extraction using a masklayer QGIS will call the command gdalwarp. This command will resample your file (it will not simply cut of some pixels at the side). Therefore it is normal that your file size will be different. You can change the resampling method with a command behind the gdal command for example "-r near" if you want to use the near resampling method.
Apparently you will not have this problem if you use the grass command r.mask.
I do not know if this is true, but here is some more info on this:
You can find more information on the resampling with gdal in the gdal documentation:
Resampling method to use. Available methods are:
nearest neighbour resampling (default, fastest algorithm, worst interpolation quality).
cubic spline resampling.
Lanczos windowed sinc resampling.
average resampling, computes the average of all non-NODATA contributing pixels. (GDAL >= 1.10.0)
mode resampling, selects the value which appears most often of all the sampled points. (GDAL >= 1.10.0)