Skip to main content
2 of 4
deleted 29 characters in body
antifrax
  • 138
  • 1
  • 7

Applying a reducer over a very large feature

The problem: How can we work around memory limits to apply a reducer over a large feature? EE's debugging page suggests either setting using tileScale to reduce the size of the tiles used for server-side parallelization if using arrays, or to set bestEffort:true with a regular feature. Neither option works in the following example.

A reproducible example: I want to use a reducer with a very large feature. For example, I want to sum the values in the white pixels that occur in the Canada polygon in the below image. Note that this polygon appears to be broken into 6 EE 'tiles'.

enter image description here

Let's work with existing EE assets:

var countries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017")
var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5")
var CAN=countries.filter(ee.Filter.eq('country_na','Canada'))

The pixels are extracted from multiband image ("hansen") at 30m resolution. Let's select one band of interest and then subset it to make the raster data as small as possible (a solution that depends on this subset is not preferred- it's specific to this dataset, and does not offer a general solution to the problem). Finally, let's convert pixel values to area, as per this tutorial.

var lossImage2016=hansen.select(['lossyear']).eq(16)
var areaImage = lossImage2016.multiply(ee.Image.pixelArea())

Running a reducer on 'areaImage' throws a memory limit error, so let's see if we can reduce the size of the raster data even further by masking 0 value pixels (this also produces the above map):

var areaImageMasked=areaImage.updateMask(areaImage)
Map.addLayer(CAN,{},'Canada') //make the map
Map.addLayer(areaImageMasked,{},'areaImageMasked')

OK, now let's try to sum the area lost:

var stats = areaImageMasked.reduceRegion({
  reducer: 'sum', //variously, 'sum' or 'ee.Reducer.sum()' are used in ee documentation
  geometry: CAN.geometry(), //is the ".geometry() doing anything?
  maxPixels: 1e13,
  scale:30, //scale of image pixel size should always be specified, as per guides/reducers_reduce_region
  bestEffort:true,
  tileScale:16 //valid tileScale is 1 to 16. Here I reduce tile size (For server-side parallelization) by 16x
});
print('pixels representing loss: ', stats.get('loss'), 'square meters')

This throws 'ComputedObject (Error): User memory limit exceeded'. Because there's so few pixels to count, I assume this is because of the size and complexity of the feature. Note that I am using the simplified version of the country boundary vector data that is preloaded in EE (and is presumably preferred and used often). So that that's odd...and suggests I'm doing something wrong!

I have identified the following sub-questions which may be helpful in finding an answer:

  • Can this problem be overcome by exporting? (I think "no" - this is a memory issue, not a computation time issue).
  • When tileScale is maxed out (at 16x, so here there are 96 tiles), how can we get EE to take it easy and run the task in even smaller digestible pieces?
  • Does changing the scale change the output of a reducer? If so, how do we quantify this?
  • Is it better to set scale (here @30 (meters), the pixel size of the 'hansen' data) or bestEffort? This source says "GEE will run your computations at the resolution of your current map view in the code editor unless you tell it otherwise. Whenever possible, explicitly set the scale arguments to force GEE to work in a scale that makes sense for your imagery/analysis".
antifrax
  • 138
  • 1
  • 7