0

I am trying to generate cloud free Sentinal 2 images, I have a basic cloud detection code working but it runs very slowly (most likely due to nested map and reduce features). I was wondering if there was a way to do this better?

function ratioRelevent(im){

// Area of the image tile
  var ftImg = ee.Algorithms.GeometryConstructors.Polygon( ee.Geometry( im.get('system:footprint') ).coordinates() )
// Intersection of the image and the ROI
  var inter = roi.intersection( ftImg, ee.ErrorMargin(0.5) )
// Percentage of the image covered by the ROI
  var coveragePer = ee.Number(inter.area()).divide(roi.area())

  var roiImgCloud = im.select(["cloud_mask", "B2"])
        .reduceToVectors({
          reducer: ee.Reducer.mean(), 
          geometry: roi, 
          scale: 100,
          geometryType: 'polygon', 
          eightConnected: false, 
          labelProperty: 'constant', 
          }).filter( ee.Filter.eq('constant', 0) )

  var addArea = function(feature) {
     return feature.set({areaHa: feature.geometry().area(ee.ErrorMargin(0.5))});
     }

  var roiImgCloudArea = roiImgCloud.map(addArea)

// the sum area of the cloud
  var areaSum = ee.Number(roiImgCloudArea.reduceColumns({
      reducer: ee.Reducer.sum(),
      selectors: ['areaHa']
      })) 

  var areaSumNum = ee.Dictionary(areaSum).get('sum')

  var CloudPerc = ee.Number(areaSumNum).divide(inter.area())
//the cloud percentage of an image WRT the ROI
  var CLOUDY_PERCENTAGE_ROI =  ee.Number(areaSumNum).divide( roi.area()) ;

  im=im.set('CLOUDY_PERCENTAGE', CloudPerc)
  im=im.set('cover_ROI', coveragePer)
  im=im.set('CLOUDY_PERCENTAGE_ROI', CLOUDY_PERCENTAGE_ROI)
  return im
}

closed as too broad by Kersten, nmtoken, whyzar, MaryBeth, BERA Feb 28 '18 at 14:27

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • For code review of working but slow code there is the Code Review Stack Exchange. – PolyGeo Feb 27 '18 at 20:15
  • If you are trying to mask out clouds, how do you get the cloud_mask band in var roiImgCloud = im.select(["cloud_mask", "B2"])? – Rodrigo E. Principe Feb 28 '18 at 2:06
  • Are you trying to calculate the cloud cover within a ROI based on the cloud masks delivered in the quality band or are you implementing your own cloud masking algorithm? What do you define as "slow" - are your exports failing or is there some other form of error? – Kersten Feb 28 '18 at 9:46
  • This band is calculated by an algorithm detailed sentinel-hub.github.io/custom-scripts/sentinel-2/… – System123 Feb 28 '18 at 9:46
2

I figured out the trick is to use Image.pixelArea() to calculate the are of the mask, rather than converting it to features and then using multiple maps and reduce functions. The new code below runs significantly faster and provides near identical output.

function calcCloudStats(img) {
   var imgPoly = ee.Algorithms.GeometryConstructors.Polygon( 
        ee.Geometry( img.get('system:footprint') ).coordinates() 
        )
   var intersection = roi.intersection(imgPoly, ee.ErrorMargin(0.5))


    var areaImg = img.select('mask').multiply(ee.Image.pixelArea())

    var stats = areaImg.reduceRegion({
       reducer: ee.Reducer.sum(),
       geometry: roi,
       scale: 10,
       maxPixels: 1e9
    })

    var cloudPercent = ee.Number(stats.get('mask')).divide(imgPoly.area())
    var coveragePercent = ee.Number(intersection.area()).divide(roi.area())
    var cloudPercentROI = ee.Number(stats.get('mask')).divide(roi.area())

    img=img.set('CLOUDY_PERCENTAGE', cloudPercent)
    img=img.set('ROI_COVERAGE_PERCENT', coveragePercent)
    img=img.set('CLOUDY_PERCENTAGE_ROI', cloudPercentROI)

    return(img)
}
  • @kersten, I've seen pixelArea() in many scripts, but as far as I know (can be easily checked) that function returns different values depending on the Map's zoom, how do you manage that? Why not using just the known area? spatial resolution is known – Rodrigo E. Principe Feb 28 '18 at 15:05
  • @RodrigoE.Principe not my answer. But I would suspect pixelArea() is calculated at the end based on the export resolution - before that it is just another node in the computational graph. So it will change with the selected export resolution but should be correct nonetheless. – Kersten Feb 28 '18 at 15:16
  • sorry @Kersten, you just edited it! my mistake. You are write, it is part of the computational node graph. I guess it is calculated in the reduceRegion function when the scale is setted. – Rodrigo E. Principe Feb 28 '18 at 15:30
  • @RodrigoE.Principe How would you calculate the area of the "1" pixels in the mask? I thought this was the reason for pixelArea (to account for the area of each pixel). This code works as expected, as I think it is only evaluated when I export the image. – System123 Mar 1 '18 at 8:05
  • @System123 I would do something like var areaImg = ee.Image.constant(100).reproject(img.projection()), but pixelArea() may be better. I made a test for trying to understand better how it works ( code.earthengine.google.com/42bb1de6922e20c042d9d3479ac7d437) but I got more confused =S – Rodrigo E. Principe Mar 1 '18 at 13:03
-2

Instead of writing your own code have you tried to use Fmask software?

Fmask 4.0 software is almost done. This version will have much better cloud, cloud shadow, and snow detection results for Sentinel-2 data and better results (compared to the 3.3 version that is currently being used by USGS as the Colection 1 QA Band) for Landsats 4-8 data as well.

  • This will not improve the runtime or the code results in any shape or form as the poster already has a working cloud-masking algorithm. – Kersten Feb 28 '18 at 9:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.