0

I am trying to calculate the total monthly precipitation rate using the Rainf_f_tavg band of the GLDAS-2.1 image collection on Google Earth Engine (https://developers.google.com/earth-engine/datasets/catalog/NASA_GLDAS_V021_NOAH_G025_T3H#bands). This is the code I have so far. However I keep getting the error user memory limit exceeded. I assume this happens because I am trying to analyse images over such a long timeframe (2001-2020), but I still get the error even when I am analysing a much shorter time frame. I can only analyse 1 single year without getting an error. I used the code in the answer to this question to try and reduce the images that need to be analysed by first computing the daily means Download GLDAS precipitation monthly 1 km resolution data by Earth-engine but it still seems to be too much. Is there any way around this or a way that I can reduce the number of images that need to be analysed further? Or is there a similar dataset I could use which is less computationally intensive?

var startDate = ee.Date('2001-01-01');
var endDate = ee.Date('2020-12-31');
var startYear = startDate.get('year');
var endYear = endDate.get('year');

var years = ee.List.sequence(startYear, endYear);
var months = ee.List.sequence(1,12);

var lst = ee.ImageCollection("NASA/GLDAS/V021/NOAH/G025/T3H")
  .select('Rainf_f_tavg')
  .filterDate(startDate, endDate);

print(lst.size());

var days = endDate.difference(startDate, 'days');
var daysStep = 1;
var timeSeries = ee.ImageCollection(
  ee.List.sequence(0, days.subtract(1), daysStep)
    .map(function (offsetDays) {
      var start = startDate.advance(offsetDays, 'days');
      var end = start.advance(daysStep, 'days');
      var composite = lst
        .filterDate(start, end)
        .mean(); // You need to decide how to combine the images
      return composite
        .set('system:time_start', start.millis())
        .set('empty', composite.bandNames().size().eq(0));
    })
  ).filterMetadata('empty', 'equals', 0);

print(timeSeries.size());

var transformed = timeSeries.map(function (image) {
  var date = image.get('system:time_start');
  var img = image.multiply(86400).set('system:time_start', date);
  return img;
});

//print(transformed);

// make monthly mean mosaics
// loop over the years and months to get summed monthly images
var byMonth = ee.ImageCollection(ee.FeatureCollection(years.map(function(y){
  var yearCollection = transformed.filter(ee.Filter.calendarRange(y, y, 'year'));
  var byYear = ee.ImageCollection.fromImages(
    months.map(function(m) {
      var maxImage = yearCollection.filter(ee.Filter.calendarRange(m, m, 'month'))
        .reduce(ee.Reducer.sum()); 
      var date = ee.Date.fromYMD(y, m, 1).format("MM_dd_YYYY");
      return maxImage.set('system:time_start', ee.Date.fromYMD(y, m, 1)).rename(date);
      //.set('month', m).set('year', y); // eventually set year and month 
  }));
  return byYear;
})).flatten());


// filter the empty one out
var outputMonthly = byMonth.filter(ee.Filter.listContains('system:band_names', 'constant').not())
                    .sort('system:time_start').toBands();

var features = outputMonthly.reduceRegions(donetsk, ee.Reducer.first(), 30);
//print(features);

Export.table.toDrive(features,
"Donetsk_monthly_precip_totals_2000_2005_v21");
1
  • I solved your issue with another approach in GEE (by using ee.List objects) with one script which spent 3 minutes for exporting aggregate values (monthly, yearly) in range '2001-01-01', '2020-12-31'. If you are still interested in your script, try out of commenting all print statements before running your script and your task.
    – xunilk
    Feb 26, 2021 at 0:24

1 Answer 1

1

When I have dealt with this kind of Image Collections, with 8 or 24 images per day, I found out that the processing is very demanding of time for monthly and yearly aggregation. Additionally, it produces a composition of images valid for a resolution of 1 arc degree instead of original 0.25 arc degrees.

On the other hand, your range of dates ('2001-01-01', '2020-12-31') includes 58,413 images from the GLDAS Image Collection and you need an efficient script for this task because it is a very big number for Google Server. So, in this case, it is preferable to extract the 58,413 values at once, mapping over them for slicing in groups of 8 elements for determining its mean (reducing them to only 7302 daily values) and, finally, pairing them with dates values before exporting to Google Drive. I have a poor Internet Service and, in my case, I spent only 9 minutes for exporting daily values to my Google Drive for your range of dates for an arbitrary point in USA. Afterward, this CSV file can be processed in a spreadsheet for monthly or yearly aggregation (or it can be used a Python Script for speeding up this process).

Complete script looks as follows:

var startDate = ee.Date('2001-01-01');
var endDate = ee.Date('2020-12-31');

var pt = ee.Geometry.Point([-101.1765625, 37.81067892891743]);

var lst = ee.ImageCollection("NASA/GLDAS/V021/NOAH/G025/T3H")
  .select('Rainf_f_tavg')
  .filterDate(startDate, endDate);
  
//print(lst);

var getPrecipit = function(image) {

  // Reducing region and getting value
  var value_precipit = ee.Image(image)
    .multiply(86400)
    .reduceRegion(ee.Reducer.first(), pt)
    .get('Rainf_f_tavg');

  return value_precipit;
};

var count = lst.size();

print("count:", count);

var precipit_list = lst.toList(count).map(getPrecipit);

//print("precipit list", precipit_list);

var len = precipit_list.size();

var list = ee.List.sequence(0, len.subtract(1), 8);

var mean_precipit_list = list.map(function(ele){

  var start = ee.Number(ele).int(); 
  var end = ee.Number(ele).add(8).int(); 

  var new_list = ee.List([]);
  var element = ee.List(precipit_list.slice(start, end)).reduce(ee.Reducer.mean());

  new_list = new_list.add(element);

  return new_list;

}).flatten();

var allDates = ee.List(lst.aggregate_array('system:time_start'));

var allDatesSimple = allDates.map(function(date){
  return ee.Date(date).format().slice(0,10);
  }).distinct();

var paired = allDatesSimple.zip(mean_precipit_list);

//print (paired);

var myFeatures = ee.FeatureCollection(paired.map(function(el){
  el = ee.List(el); // cast every element of the list
  var geom = pt;
  return ee.Feature(geom, {
    'date': ee.String(el.get(0)),
    'value':ee.Number(el.get(1))
  });
}));

//print(myFeatures);

// Export features, specifying corresponding names.
Export.table.toDrive(myFeatures,
"precipitation", //my task
"GEE_Folder", //my export folder
"dayly_precipit",  //file name
"CSV");

Obtained CSV file can be downloaded from here.

2
  • Hey @xunilk, i tried to use use your script where I changed the geometry with feature collections with multiple polygons, but I got this error. ==> Error: Unable to use a collection in an algorithm that requires a feature or image. This may happen when trying to use a collection of collections where a collection of features is expected; use flatten, or map a function to convert inner collections to features. Use clipToCollection (instead of clip) to clip an image to a collection. Jun 11, 2021 at 19:18
  • Is there any way possible to download the long term data with multiple polygon/assets of shapefile. The source script is at code.earthengine.google.co.in/… and the shapefile is code.earthengine.google.co.in/?asset=users/abhilashaanu92/… Jun 11, 2021 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.