0

I tried running my code without exporting to drive and it works fine with no errors. But as soon as I add in the block of code that exports images to drive, I get a bunch of messages saying "Earth Engine memory exceeded." However, it still outputs many of the images but since there are so many it's hard to tell if some are getting skipped due to the error. Is there any way to refactor my code to solve this issue?

Context: I am trying to get an NDVI image for each listed country for every two-year period between 1985 and 2012. I aim to only obtain an image from the summer months. I start by generating an image from just the month of July but if the country's image is missing more than 5% of its pixels, I expand the month-range and re-evaluate (hence why month_ranges is ordered and I use .iterate on it). If it is expanded all the way to May-October and still missing more than 5%, we set it back to a default month range of June-September.

I also originally had years as an ee.List for which I also mapped over it, but I started getting the "too many concurrent aggregations" error and that was fixed when I changed years to a primitive list and used forEach().

// ------------- Helper Functions ------------------
// helper masking function for extracting variable of interest from a specific bitMask band
var getQABits = function(image, start, end, newName) {
    var pattern = 0;
    for (var i = start; i <= end; i++) {
       pattern += Math.pow(2, i);
    }
    return image.select([0], [newName]) // selects the first band and renames it to newName
                  .bitwiseAnd(pattern) // output number has 1 for each bit if both input numbers (band and pattern) have a 1 for that bit, otherwise bit is 0
                  .rightShift(start);
};

// function that returns a binary mask where each pixel is 1 if it is not a cloud shadow, otherwise 0.
var cloud_shadows = function(image) {
  var QA = image.select(['pixel_qa']);
  return getQABits(QA, 3,3, 'Cloud_shadows').neq(1);
};

// function that returns a binary mask where each pixel is 1 if it is not a cloud, otherwise 0.
var clouds = function(image) {
  var QA = image.select(['pixel_qa']);
  return getQABits(QA, 5,5, 'Cloud').neq(1);
};

// function that returns a binary mask where each pixel is 1 if it is not water, 0 if it is.
var water = function(image) {
  var QA = image.select(['pixel_qa']);
  return getQABits(QA, 2,2, 'Water').neq(1);
};

// updates the image so that it drops pixels that are cloud shadows, clouds, or water
var maskClouds = function(image) {
  var b = water(image);
  var cs = cloud_shadows(image);
  var c = clouds(image);
  var image1 = image.updateMask(cs);
  image = image1.updateMask(b);
  return image.updateMask(c);
};

// returns the NDVI values of an image
var NDVI = function(image) {
  return image.normalizedDifference(['B4', 'B3']).rename('NDVI');
};

// fraction pixel coverage
function coverage(image, roi) {
  var mask = image.mask();
  var frac = mask.reduceRegion({
    reducer: ee.Reducer.mean(),
    geometry: roi,
    scale: 500,  // Adjust the scale as needed
    maxPixels: 1e19,
  });
  return ee.Number(frac.get('NDVI'));
}



// ----------------- Main --------------------------

var countryGeos = ee.FeatureCollection('USDOS/LSIB_SIMPLE/2017');
var landsat = ee.ImageCollection("LANDSAT/LT05/C01/T1_SR");


// countries
var countries = ee.List([
  "Albania",
  "Austria",
  "Belgium",
  "Bosnia & Herzegovina",
  "Bulgaria",
  "Croatia",
  "Czechia",
  "Denmark",
  "Estonia",
  "Finland",
  "France",
  "Germany",
  "Greece",
  "Hungary",
  "Ireland",
  "Italy",
  "Kosovo",
  "Latvia",
  "Lithuania",
  "Luxembourg",
  "Macedonia",
  "Montenegro",
  "Netherlands",
  "Norway",
  "Poland",
  "Portugal",
  "Romania",
  "Serbia",
  "Slovakia",
  "Slovenia",
  "Spain",
  "Sweden",
  "Switzerland",
  "United Kingdom"
]);

// 2-year period
//var years = ee.List.sequence(1985, 2012, 2)
var years = []
for (var y = 1985; y < 2012; y += 2){
  years.push(y)
}

// month range
var month_ranges = ee.List([[7, 7], [7, 8], [6, 8], [6, 9], [5, 9], [5, 10], [6, 8]]);



years.forEach(function(year1){
  year1 = ee.Number(year1).uint16()
  var year2 = year1.add(1).uint16()
  
  // filter landsat
  var start_date = ee.Date.fromYMD(year1, 1, 1);
  var end_date = ee.Date.fromYMD(year2, 12, 31)
  var landsat_2yr = landsat.filterDate(start_date, end_date)
  
  var outputCollection = ee.ImageCollection(countries.map(function(country){
    // always cast arg if server-side object
    country = ee.String(country)
    
    // get region of interest and filter landsat further
    var roi = countryGeos.filter(ee.Filter.eq('country_na', country)).geometry();
    var landsat_country = landsat_2yr.filterBounds(roi)
    
    // return the best image by evaluating month ranges in order
    var first = ee.Image().set('coverage', 0)
    return month_ranges.iterate(function(cur_mr, prevResult) {
      // always cast arg
      prevResult = ee.Image(prevResult)
      
      // filter landsat to month range
      var month1 = ee.List(cur_mr).get(0)
      var month2 = ee.List(cur_mr).get(1)
      var landsat_summer = landsat_country.filter(ee.Filter.calendarRange(month1, month2, 'month'));
      
      // reduce to one image and calculate median NDVI for this time frame
      var medianNDVI = landsat_summer.map(maskClouds).map(NDVI).median();
      
      // reclassify NDVI values to discrete values from 0 to 100
      var result = medianNDVI.multiply(100).where(medianNDVI.lte(0), -1).add(1).int();
      
      // construct properties to add to image
      var cov = coverage(result, roi)
      var filename = ee.String('NDVIreclass')
        .cat('_').cat(ee.Number(year1))
        .cat('-').cat(ee.Number(year2))
        .cat('_').cat(month1)
        .cat('-').cat(month2)
        .cat('_').cat(country)
      
      // add properties to image
      var curResult = result
        .set('filename', filename)
        .set('roi', roi)
        .set('coverage', cov)
      
      // ask if previous image is missing more than 5% of its pixels
      var condition = ee.Number(prevResult.get('coverage')).lt(0.95)
      
      // if we haven't found a satisfactory image yet, return current result
      return ee.Algorithms.If(condition, curResult, prevResult)
    },
      first
    );
    
    
  }));
  
  print(outputCollection)
  
  // ERRORS ARE INTRODUCED HERE BUT NOT CONSISTENTLY
  outputCollection.aggregate_array('filename')
  .evaluate(function(filenames) { // error is specifically pointing here
    filenames.forEach(function(filename) {
      var image = outputCollection
        .filter(ee.Filter.eq('filename', filename))
        .first()
        
      Export.image.toDrive({
        image: image,
        description: filename,
        scale: 300,
        maxPixels: 3784216672400,
        folder : 'algorithmic-ndvi',
        region: ee.Geometry(image.get('roi'))
      })
    })
  })
})

1 Answer 1

1

Using foreach is no better than using a for loop, and you usually don't want to do that either for reasons exactly like this. In this case, though, because you're ultimately trying to Export a bunch of individual images, one or the other can't be helped. That said, I'd build this somewhat differently to avoid the foreach-map-iterate-foreach nested loops that are mixing client and server side stuff.

  1. Use 3 nested maps to build a collection of year/country/months combinations (think of these as formulas).
  2. For each formula, build the corresponding cloud-masked collection (create a function that takes year, region and start/end months) and run the coverage reduceRegion on a composite made from that. Note you could skip all the NDVI stuff here; you just need to get the coverage of one of the bands. You could use the coverage of the QA band you're cloud masking with; then you would only need 1/3 as much input data and will run faster and use less memory.
  3. Export the reduceRegion results as a table with the year+country+months parameters used to make each one. This will take a while (but you could probably run this at a lower resolution and be fine; that would speed it up).

Everything up to here doesn't need any looping; it can all be done with server-side map functions and one table export.

  1. Load the result table from step 3.

  2. Do a client-side double for-loop over year+country. In each loop:

  3. filter the table for year+country, find the "best" coverage answer over the months for that region/time range (via max reducer?) and (re)generate just that composite's median. Then compute NDVI results you want, clip and export.

5
  • Hey Noel, thanks for commenting. I tried to implement this but I didn't get far before running into errors. I built a list of dictionaries per point 1 where each dictionary has keys "year", "country", and "month_range". Then, per point 2, I made a function that returns an image collection per your description. However, when mapping over formulas and using the function to build the collection and then using coverage to reduce it (also changed to use pixel_qa instead of NDVI), I run into "User memory limit exceeded". Am I deviating from what you were thinking of doing?
    – epsilon
    Commented Nov 1, 2023 at 20:21
  • Here is the function for reference: // build image collection given parameters function buildCollection(year, roi, month_range){ year = ee.Number(year); month_range = ee.List(month_range); var start_date = ee.Date.fromYMD(year, 1, 1); var end_date = ee.Date.fromYMD(year.add(1), 12, 31); return landsat .filterBounds(roi) .filterDate(start_date, end_date) .filter(ee.Filter.calendarRange(month_range.get(0), month_range.get(1), 'month')) .map(maskClouds); }
    – epsilon
    Commented Nov 1, 2023 at 20:42
  • Normally, you'd want to answer code pasted in, but it's huge, and isn't the important part. code.earthengine.google.com/9e50517724b563a2f6cc6e873b500bf0 Commented Nov 2, 2023 at 21:42
  • Oh, wait... you probably don't do the .bounds() on the geometry that you pass to coverage, it'll produce a skewed answer. But you do want it on the filterBounds, for memory reasons. Commented Nov 2, 2023 at 22:35
  • Thanks for the suggestion, I got pretty far with it but unfortunately am hitting a wall again near the end of the pipeline. I split the whole process it into two scripts. One exports a table, the other imports the table and finds the best images and tries to export them. It exports fine when using a small year range but when scaling up, I get an error: "cannot read property 'forEach' of undefined". Is there anything I'm doing wrong here? script 1: code.earthengine.google.com/a3b3d083783225d3a5147b2971f540a5 script2: code.earthengine.google.com/d10dc09586be01d5d2cfd451c648a489
    – epsilon
    Commented Nov 9, 2023 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.