I suppose your pixel size is 10m². You could use the gdal sieve function you can find in the Raster menu to remove lonely pixels. You can also launch it by searching in the command line :
Processing algorithm: Sieve
Details : http://www.gdal.org/gdal_sieve.html
The gdal_sieve.py script removes raster polygons smaller than a
provided threshold size (in pixels) and replaces replaces them with
the pixel value of the largest neighbour polygon. The result can be
written back to the existing raster band, or copied into a new file.
I thing the polygon word is not really appropriate but the tool should do what you want without going through heavy polygon vectorisation. You can raise the value to remove bigger areas or should your pixel size was 1m² for example. You would than use 10 as a threshold.
For french readers, the tool is called "Tamiser" but the tool within gdal is still called Sieve.