Cloud Mask Landsat 7 Tier 2 TOA

Question

I am trying to create a decent cloudfree mosaic of Landsat 7 Tier 2 TOA reflectance around the Amery Iceshelf in Antarctica. I did find a good script for landsat 8 TOA (Apply a cloud mask to a Landsat8 collection in Google Earth Engine - time series) and those work for me, however when I try to change this to landsat 7 with the correct parameters, it doesn't.

Next to the couldmask issue, the result is a collection, whereas I would like to have 1 (mosaiced) image. Any idea?

This is the code I am using at the moment:

var geometry = /* color: #00ffff */ee.Geometry.Polygon(
        [[[92.57694067490729, -70.98282239796863],
          [57.06912817490718, -70.57777747363254],
          [51.35623754990718, -76.61497412799037],
          [91.96170629990729, -77.28946992862909],
          [92.66483129990729, -71.18228294956876]]]);



// TOA L7 dataset
var mosaic =  ee.ImageCollection("LANDSAT/LE07/C01/T2_TOA")
                  .filterDate('1999-12-01', '2003-03-01')
//                  .filterMetadata('CLOUD_COVER', 'less_than',10)
                  .filterBounds(geometry)
 //                 .sort('CLOUD_COVER',false)
//                  .first()

 var getQABits = function(image, start, end, newName) {
    // Compute the bits we need to extract.
    var pattern = 0;
    for (var i = start; i <= end; i++) {
       pattern += Math.pow(2, i);
    }
    // Return a single band image of the extracted QA bits, giving the band
    // a new name.
    return image.select([0], [newName])
                  .bitwiseAnd(pattern)
                  .rightShift(start);
};

// A function to mask out cloudy pixels.
var cloud_shadows = function(image) {
  // Select the QA band.
  var QA = image.select(['BQA']);
  // Get the internal_cloud_algorithm_flag bit.
  return getQABits(QA, 7,8, 'Cloud_shadows').eq(1);
  // Return an image masking out cloudy areas.
};

// A function to mask out cloudy pixels.
var clouds = function(image) {
  // Select the QA band.
  var QA = image.select(['BQA']);
  // Get the internal_cloud_algorithm_flag bit.
  return getQABits(QA, 4,4, 'Cloud').eq(0);
  // Return an image masking out cloudy areas.
};

var maskClouds = function(image) {
  var cs = cloud_shadows(image);
  var c = clouds(image);
  image = image.updateMask(cs);
  return image.updateMask(c);
};

var mosaic_free = mosaic.map(maskClouds);
var visParamsPanSharp = {bands: ['red', 'green', 'blue'],min: [0,0,0],max: [1, 1, 1]};


Map.addLayer(mosaic, visParams, 'With clouds'); 
Map.addLayer(mosaic_free, visParams, 'Cloud free'); 

Answer 1

Firstly, I have to point out that on brights surfaces as snow and bare sand the cloudmask algorithm doesn't perform well. You can find that easily and quickly on the NASA site. See for example here

Next, if you make a mosaic over sufficient images, you will end up with pixels you prefer and so the mosaicing itself filters out the clouds if you would do the mosaic correctly. After a bit of research, it seems that clouds are higher in SWIR bands, so reversing the band order of a SWIR band and applying that as input to the quality mosaic will filter out clouds. Unfortunately, both shadow and water are lower in SWIR values, so you will end up with lots of shadow pixels. What seems to perform better is apply the quality mosaic on the thermal infrared band, as clouds a relatively colder that the surface. Here is the code I used. You will have to play around with it to get best results. You could maybe filter out winter or summer images or apply the quality mosaic on different (combinations of) bands:

var geometry = /* color: #00ffff */ee.Geometry.Polygon(
    [[[92.57694067490729, -70.98282239796863],
      [57.06912817490718, -70.57777747363254],
      [51.35623754990718, -76.61497412799037],
      [91.96170629990729, -77.28946992862909],
      [92.66483129990729, -71.18228294956876]]]);

// TOA L7 dataset
var imageCollection =  ee.ImageCollection("LANDSAT/LE07/C01/T2_TOA")
              .filterDate('1999-12-01', '2003-03-01')
              .filterBounds(geometry);

// Reverse the values of the SWIR band for the quality mosaic
var imageCollection = imageCollection.map(function(image){
  return image.addBands(image.select('B5').subtract(1).abs().rename('QAband'));
});

// apply the quality mosaic for the SWIR and TIR bands
var mosaic_SWIR = imageCollection.qualityMosaic('QAband');
var mosaic_TIR = imageCollection.qualityMosaic('B6_VCID_1');

// Visualize the results
var visParams = {bands: ['B4', 'B3', 'B2'],min: 0, max:1};
Map.addLayer(mosaic_SWIR, visParams, 'With SWIR'); 
Map.addLayer(mosaic_TIR, visParams, 'With TIR'); 
Map.centerObject(geometry)

Answer 2

I'm a bit late to this party, but I had a similar need and notice that you are using the wrong bits for LS7. Have a look at this link in the GEE documentation from which the code snippet below is taken:

var cloudMaskL457 = function(image) {
  var qa = image.select('pixel_qa');
  // If the cloud bit (5) is set and the cloud confidence (7) is high
  // or the cloud shadow bit is set (3), then it's a bad pixel.
  var cloud = qa.bitwiseAnd(1 << 5)
                  .and(qa.bitwiseAnd(1 << 7))
                  .or(qa.bitwiseAnd(1 << 3));
  // Remove edge pixels that don't occur in all bands
  var mask2 = image.mask().reduce(ee.Reducer.min());
  return image.updateMask(cloud.not()).updateMask(mask2);
};

This explains why you are getting a strange result, because the relevant bits for LS8 are not identical for LS7. I have used this in my project area and it works well.