A mask is a shape or set of values that includes or excludes portions of data.
In GIS, masks are typically polygon shapes. A particular operation utilizes the mask shape when performed on another set of data to only operate on the area either within or outside of the mask, depending on implementation. For example, one could use a polygon outline of a lake as a mask when computing statistics on a set of points - those points within the lake would not be included in the statistic calculations.
Masks can also be numeric or value based, rather than shapes. For example in a raster DEM, one can mask out all cells less than x, resulting in a raster with either 0 or nodata for those cells while cells greater than x retain their original values.
Masks can be created as static entities (a polygon or binary raster) or created on-the-fly via a calculation or query.