Return to Question

• here (OP asked basically the same question; it has been ignored)
• here (OP wanted to use reduce region on a feature collection where memory / time limits were a risk - neither issue arose, and no workarounds are provided)
• here (OP wanted to filter and sort an image collection for an arbitrary (and large) number of vector geometries, and export the result. The discussion appears to show how to set up such a script, but doesn't address memory or computation time limits)
• here (OP mapped a reducer over a feature collection, but did not discuss memory or time limits)
• here (OP is using the python API and iterating with a for() loop; memory or time limits don't come up).
• and here (OP is iterating over years in the same example dataset used in this question; memory and time limits don't come up)

• here (OP asked about memory limits when mapping over a large feature collection and was directed to seek help in a closed google group)
• here (OP asked basically the same question; it has been ignored)
• here (OP wanted to use reduce region on a feature collection where memory / time limits were a risk - neither issue arose, and no workarounds are provided)
• here (OP wanted to filter and sort an image collection for an arbitrary (and large) number of vector geometries, and export the result. The discussion appears to show how to set up such a script, but doesn't address memory or computation time limits)
• here (OP mapped a reducer over a feature collection, but did not discuss memory or time limits)
• here (OP is using the python API and iterating with a for() loop; memory or time limits don't come up).
• and here (OP is iterating over years in the same example dataset used in this question; memory and time limits don't come up)

The problem: I want to iterate a simple calculation (e.g. a Reducer, or a simple user-defined function) over a very large feature collection. The task either exceeds EE's memory limits (throwing a 'memory limits exceeded' error), or takes too long (throwing a 'computation timed out error'). How can the task best be broken down to avoid these problems?

Related questions have been asked

• here (OP wanted to use reduce region on a feature collection where memory / time limits were a risk - neither issue arose, and no workarounds are provided)
• here (OP wanted to filter and sort an image collection for an arbitrary (and large) number of vector geometries, and export the result. The discussion appears to show how to set up such a script, but doesn't address memory or computation time limits)
• here (OP mapped a reducer over a feature collection, but did not discuss memory or time limits)
• here (OP is using the python API and iterating with a for() loop; memory or time limits don't come up).
• and here (OP is iterating over years in the same example dataset used in this question; memory and time limits don't come up)

The problem can be demonstrated using pre-loaded EE assets. Let's try to use vector dataset (feature collection) of the world's national boundaries to count the area of forest loss recorded by Hansen et al 2013 (update), for each country in the world:

var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5");
var countries = ee.FeatureCollection("USDOS/LSIB/2013");
//this is a detailed map of country boundaries for the whole world
var simpleCountries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017");
//the same thing, but simplified. Small islands and other complicated vector features do not appear
var fourCountries = countries.filter(ee.Filter.inList('name',['SLOVAKIA','BURKINA FASO','BHUTAN','CONGO (Brazzaville)']));
//this is a test collection of four small countries, with relatively simple geometry, plus Congo (to check vs 'Congo' result [in ee tutorial][5])

var lossImage=hansen.select(['loss']) //all Hansen 'loss' pixels, 2000-2017
//these are 30m pixels showing 'forest loss', which we will try to count
var lossIn2012=hansen.select(['lossyear']).eq(12)
//to make things easy, we will work with just one year's loss data at a time
var areaImage=lossIn2012.multiply(ee.Image.pixelArea())
//create a layer where values are pixel area


//the following call (with just 4 small countries) succeeds in less than a minute
//the area reported for Congo (in 2012) accords with the area in the tutorial. Nice!
var smallReduction=areaImage.reduceRegions({
  collection:fourCountries,
  reducer:'sum'
})
print('smallReduction', smallReduction)

//this call fails ("computation timed out")
var hugeReduction=areaImage.reduceRegions({
  collection:countries,
  reducer:'sum'
})
print('hugeReduction', hugeReduction)

//following [the debugging guide][5], we can use map with 
//reduceRegion to break the reduction into smaller tasks.
//this call also fails ("computation timed out")
var mappedReduction=countries.map(function(feature){
  return feature.set(areaImage.reduceRegion({
    reducer:'sum',
    geometry:feature.geometry(),
    maxPixels:1e15 //default is 1e9, which is exceeded by an early country in the list (Brazil?), throwing an error
  }))
})
print('mappedReduction', mappedReduction)

//following debugging guide [again][5] we can export the result...this doesn't get us more memory, but
//will stop the computation from timing out
//still running @ >5 hours.....
Export.table.toDrive({
  collection:mappedReduction,
  description:'mappedReduction',
  fileFormat:'CSV'
})

What gives? Yes, this is a whole lot of pixels - but we're here (maybe learning JavaScript) on the promise of 'planetary scale analysis'. How can we run planetary-scale analyses in a way that jives with the resources the wonderful folks are Google are able to make available?

Specifically:

• Can we use a feature collection of smaller features, e.g. sub-national boundaries? (for example, I tried to load the GADM sub-national shapefiles, level 1, but no dice - some features in this file had vertices above the allowed limit of 1 million)
• If we use the bestEffort or tileScale parameters in the call to reduceRegion, how can we make the trade-off in accuracy verbose? How can we 'see' what sort of aggregation and scaling is going on to simplify the computation?
• If iterating over a large feature collection, can export be used to create a seperate file for each feature, so we can see the progress of a multi-hour operation? (is there another way to check on the status of a task?)
• If a single country is too big to use reduceRegion with our dataset (for example, Russia, with the 30m Hansen pixels) how can such a country be broken into smaller units that ee CAN process? Can this be automatically embedded in the map call?

Lastly, the general question: what's the 'EE way' to handle this sort of very large analysis? Slow (even multi-day) is OK - it just needs to run. Notice that here I'm working with one year of the Hansen data, and not trying to iterate over years. It should be possible to spit out a count of area deforested per country per year .... shouldn't it?

The problem: What is the best way to iterate a simple calculation (e.g. a Reducer, or a simple user-defined function) over a very large feature collection? In this context, 'best' means (1) the computation works (does not throw either a 'memory limits exceeded' or a 'computation timed out error'), and (2) works fast and in EE way (<=> does not consume server-side resources unnecessarily).

A list of related-but-not-useful questions found to date is appended.

A reproducible example: The problem can be demonstrated using pre-loaded EE assets. Let's try to iterate over a vector dataset (an EE 'feature collection') of the world's national boundaries in order to count the area of forest loss recorded by Hansen et al (2017 update) for each country in the world:

var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5");
var countries = ee.FeatureCollection("USDOS/LSIB/2013");
//this is a detailed map of country boundaries for the whole world
var simpleCountries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017");
//the same thing, but simplified. Small islands and other complicated vector features do not appear
var fourCountries = countries.filter(ee.Filter.inList('name',['SLOVAKIA','BURKINA FASO','BHUTAN','CONGO (Brazzaville)']));
//this is a test collection of four small countries, with relatively simple geometry, plus Congo (to check vs 'Congo' result [in ee tutorial][5])

var lossImage=hansen.select(['loss']) //all Hansen 'loss' pixels, 2000-2017
//these are 30m pixels showing 'forest loss', which we will try to count
var lossIn2012=hansen.select(['lossyear']).eq(12)
//to make things easy, we will work with just one year's loss data at a time
var areaImage=lossIn2012.multiply(ee.Image.pixelArea())
//create a layer where values are pixel area


//the following call (with just 4 small countries) succeeds in less than a minute
//the area reported for Congo (in 2012) accords with the area in the tutorial. Nice!
var smallReduction=areaImage.reduceRegions({
  collection:fourCountries,
  reducer:'sum'
})
print('smallReduction', smallReduction)

//this call fails ("computation timed out")
var hugeReduction=areaImage.reduceRegions({
  collection:countries,
  reducer:'sum'
})
print('hugeReduction', hugeReduction)

//following [the debugging guide][5], we can use map with 
//reduceRegion to break the reduction into smaller tasks.
//this call also fails ("computation timed out")
var mappedReduction=countries.map(function(feature){
  return feature.set(areaImage.reduceRegion({
    reducer:'sum',
    geometry:feature.geometry(),
    maxPixels:1e15 //default is 1e9, which is exceeded by an early country in the list (Brazil?), throwing an error
  }))
})
print('mappedReduction', mappedReduction)

//following debugging guide [again][5] we can export the result...this doesn't get us more memory, but
//will stop the computation from timing out
//killed it after 22 hours of runtime...
Export.table.toDrive({
  collection:mappedReduction,
  description:'mappedReduction',
  fileFormat:'CSV'
})

What gives? Yes, this is a whole lot of pixels - but we're here (maybe learning JavaScript) on the promise of 'planetary scale analysis'. So how can we run planetary-scale analyses in a way that jives with the resources the wonderful folks are Google are able to make available? I (newbie) think answering the following sub-questions might help:

• Can the calculation task (ee.Reducer.sum()) broken up or slowed down to allow the computation to succeed? [edit] I think this merits it's own question, so have opened one here. If the script in this question fails because some features are just too big, the problem
  can be overcome by using larger scale feature collection (but do see that question - EE appears to choke on features of a reasonable size for 'planetary analysis')
• If we use the bestEffort or tileScale parameters in the call to reduceRegion, how can we make the result (and any resulting trade-off in accuracy) verbose? I've played with both (no luck), but can't see what's going on ... or where they fail.
• Can export be told to create a separate output file for each feature, so we can see the progress of a multi-hour operation  (and see if it fails on a specific feature)?

Related-but-not-useful questions have been asked

• here (OP asked basically the same question; it has been ignored)
• here (OP wanted to use reduce region on a feature collection where memory / time limits were a risk - neither issue arose, and no workarounds are provided)
• here (OP wanted to filter and sort an image collection for an arbitrary (and large) number of vector geometries, and export the result. The discussion appears to show how to set up such a script, but doesn't address memory or computation time limits)
• here (OP mapped a reducer over a feature collection, but did not discuss memory or time limits)
• here (OP is using the python API and iterating with a for() loop; memory or time limits don't come up).
• and here (OP is iterating over years in the same example dataset used in this question; memory and time limits don't come up)

var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5");
var countries = ee.FeatureCollection("USDOS/LSIB/2013");
//this is a detailed map of country boundaries for the whole world
var simpleCountries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017");
//the same thing, but simplified. Small islands and other complicated vector features do not appear
var fourCountries = countries.filter(ee.Filter.inList('name',['SLOVAKIA','BURKINA FASO','BHUTAN','CONGO (Brazzaville)']));
//this is a test collection of four small countries, with relatively simple geometry, plus Congo (to check vs 'Congo' result [in ee tutorial][5])

var lossImage=hansen.select(['loss']) //all Hansen 'loss' pixels, 2000-2017
//these are 30m pixels showing 'forest loss', which we will try to count
var lossIn2012=hansen.select(['lossyear']).eq(12)
//to make things easy, we will work with just one year's loss data at a time
var areaImage=lossIn2012.multiply(ee.Image.pixelArea())
//create a layer where values are pixel area


//the following call (with just 4 small countries) succeeds in less than a minute
//the area reported for Congo (in 2012) accords with the area in the tutorial. Nice!
var smallReduction=areaImage.reduceRegions({
  collection:fourCountries,
  reducer:'sum'
})
print(smallReduction)

//this call fails ("computation timed out")
var hugeReduction=areaImage.reduceRegions({
  collection:countries,
  reducer:'sum'
})
print(hugeReduction)

//following [the debugging guide][5], we can use map with 
//reduceRegion to break the reduction into smaller tasks.
//this call also fails ("computation timed out")
var mappedReduction=countries.map(function(feature){
  return feature.set(areaImage.reduceRegion({
    reducer:'sum',
    geometry:feature.geometry(),
    maxPixels:1e15 //default is 1e9, which is exceeded by an early country in the list (Brazil?), throwing an error
  }))
})
print(mappedReduction)

//following debugging guide [again][5] we can export the result...this doesn't get us more memory, but
//will stop the computation from timing out
//still running @ >5 hours.....
Export.table.toDrive({
  collection:mappedReduction,
  description:'mappedReduction',
  fileFormat:'CSV'
})

Lastly, the general question: what's the 'EE way' to handle this sort of very large analysis? Slow (even multi-day) is OK - it just needs to run. Notice that here I'm working with one hear of the Hansen data, and not trying to iterate over years. It should be possible to spit out a count of area deforested per country per year .... shouldn't it?

var hansen = ee.Image("UMD/hansen/global_forest_change_2017_v1_5");
var countries = ee.FeatureCollection("USDOS/LSIB/2013");
//this is a detailed map of country boundaries for the whole world
var simpleCountries = ee.FeatureCollection("USDOS/LSIB_SIMPLE/2017");
//the same thing, but simplified. Small islands and other complicated vector features do not appear
var fourCountries = countries.filter(ee.Filter.inList('name',['SLOVAKIA','BURKINA FASO','BHUTAN','CONGO (Brazzaville)']));
//this is a test collection of four small countries, with relatively simple geometry, plus Congo (to check vs 'Congo' result [in ee tutorial][5])

var lossImage=hansen.select(['loss']) //all Hansen 'loss' pixels, 2000-2017
//these are 30m pixels showing 'forest loss', which we will try to count
var lossIn2012=hansen.select(['lossyear']).eq(12)
//to make things easy, we will work with just one year's loss data at a time
var areaImage=lossIn2012.multiply(ee.Image.pixelArea())
//create a layer where values are pixel area


//the following call (with just 4 small countries) succeeds in less than a minute
//the area reported for Congo (in 2012) accords with the area in the tutorial. Nice!
var smallReduction=areaImage.reduceRegions({
  collection:fourCountries,
  reducer:'sum'
})
print('smallReduction', smallReduction)

//this call fails ("computation timed out")
var hugeReduction=areaImage.reduceRegions({
  collection:countries,
  reducer:'sum'
})
print('hugeReduction', hugeReduction)

//following [the debugging guide][5], we can use map with 
//reduceRegion to break the reduction into smaller tasks.
//this call also fails ("computation timed out")
var mappedReduction=countries.map(function(feature){
  return feature.set(areaImage.reduceRegion({
    reducer:'sum',
    geometry:feature.geometry(),
    maxPixels:1e15 //default is 1e9, which is exceeded by an early country in the list (Brazil?), throwing an error
  }))
})
print('mappedReduction', mappedReduction)

//following debugging guide [again][5] we can export the result...this doesn't get us more memory, but
//will stop the computation from timing out
//still running @ >5 hours.....
Export.table.toDrive({
  collection:mappedReduction,
  description:'mappedReduction',
  fileFormat:'CSV'
})

Lastly, the general question: what's the 'EE way' to handle this sort of very large analysis? Slow (even multi-day) is OK - it just needs to run. Notice that here I'm working with one year of the Hansen data, and not trying to iterate over years. It should be possible to spit out a count of area deforested per country per year .... shouldn't it?