I'm an EE newbie, and would like to create a detailed 'resource' for the following challenge.
The problem: I want to iterate a simple calculation (e.g. a Reducer, or a simple user-defined function) over a very large feature collection. The task either exceeds EE's memory limits (throwing a 'memory limits exceeded' error), or takes too long (throwing a 'computation timed out error'). How can the task best be broken down to avoid these problems?
Related questions have been asked
- here (OP wanted to use reduce region on a feature collection where memory / time limits were a risk - neither issue arose, and no workarounds are provided)
- here (OP wanted to filter and sort an image collection for an arbitrary (and large) number of vector geometries, and export the result. The discussion appears to show how to set up such a script, but doesn't address memory or computation time limits)
- here (OP mapped a reducer over a feature collection, but did not discuss memory or time limits)
- here (OP is using the python API and iterating with a for() loop; memory or time limits don't come up).
- and here (OP is iterating over years in the same example dataset used in this question; memory and time limits don't come up)
The problem can be demonstrated using pre-loaded EE assets. Let's try to use vector dataset (feature collection) of the world's national boundaries to count the area of forest loss recorded by Hansen et al 2013 (update), for each country in the world:
var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5");
var countries = ee.FeatureCollection("USDOS/LSIB/2013");
//this is a detailed map of country boundaries for the whole world
var simpleCountries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017");
//the same thing, but simplified. Small islands and other complicated vector features do not appear
var fourCountries = countries.filter(ee.Filter.inList('name',['SLOVAKIA','BURKINA FASO','BHUTAN','CONGO (Brazzaville)']));
//this is a test collection of four small countries, with relatively simple geometry, plus Congo (to check vs 'Congo' result [in ee tutorial][5])
var lossImage=hansen.select(['loss']) //all Hansen 'loss' pixels, 2000-2017
//these are 30m pixels showing 'forest loss', which we will try to count
var lossIn2012=hansen.select(['lossyear']).eq(12)
//to make things easy, we will work with just one year's loss data at a time
var areaImage=lossIn2012.multiply(ee.Image.pixelArea())
//create a layer where values are pixel area
//the following call (with just 4 small countries) succeeds in less than a minute
//the area reported for Congo (in 2012) accords with the area in the tutorial. Nice!
var smallReduction=areaImage.reduceRegions({
collection:fourCountries,
reducer:'sum'
})
print(smallReduction)
//this call fails ("computation timed out")
var hugeReduction=areaImage.reduceRegions({
collection:countries,
reducer:'sum'
})
print(hugeReduction)
//following [the debugging guide][5], we can use map with
//reduceRegion to break the reduction into smaller tasks.
//this call also fails ("computation timed out")
var mappedReduction=countries.map(function(feature){
return feature.set(areaImage.reduceRegion({
reducer:'sum',
geometry:feature.geometry(),
maxPixels:1e15 //default is 1e9, which is exceeded by an early country in the list (Brazil?), throwing an error
}))
})
print(mappedReduction)
//following debugging guide [again][5] we can export the result...this doesn't get us more memory, but
//will stop the computation from timing out
//still running @ >5 hours.....
Export.table.toDrive({
collection:mappedReduction,
description:'mappedReduction',
fileFormat:'CSV'
})
What gives? Yes, this is a whole lot of pixels - but we're here (maybe learning JavaScript) on the promise of 'planetary scale analysis'. How can we run planetary-scale analyses in a way that jives with the resources the wonderful folks are Google are able to make available?
Specifically:
- Can we use a feature collection of smaller features, e.g. sub-national boundaries? (for example, I tried to load the GADM sub-national shapefiles, level 1, but no dice - some features in this file had vertices above the allowed limit of 1 million)
- If we use the bestEffort or tileScale parameters in the call to reduceRegion, how can we make the trade-off in accuracy verbose? How can we 'see' what sort of aggregation and scaling is going on to simplify the computation?
- If iterating over a large feature collection, can export be used to create a seperate file for each feature, so we can see the progress of a multi-hour operation? (is there another way to check on the status of a task?)
- If a single country is too big to use reduceRegion with our dataset (for example, Russia, with the 30m Hansen pixels) how can such a country be broken into smaller units that ee CAN process? Can this be automatically embedded in the map call?
Lastly, the general question: what's the 'EE way' to handle this sort of very large analysis? Slow (even multi-day) is OK - it just needs to run. Notice that here I'm working with one hear of the Hansen data, and not trying to iterate over years. It should be possible to spit out a count of area deforested per country per year .... shouldn't it?