1

On GEE, I want to export daily weather variables for 18 years for about 30,000 rectangular polygons. That would be exporting a FeatureCollection to a csv file with ~200 million rows (30000 places * 365 days * 18 years). The export job took 2 days and then failed with this error: Error: Internal error. Is there an efficient way to export such a large feature collection?

To know if size of the export was a problem, I also tried another export with bigger rectangular polygons (which numbered about 6,000), resulting in ~40 million rows. That job finished in about 2 hours. So I am not sure if size of the dataset is the problem here, or if something else is at play.

Here's how I am exporting:

// grids: import a FeatureCollection with 30,000 grids
// geometry: import a bounding box

var era5 = ee.ImageCollection('ECMWF/ERA5/DAILY')
                   .select(['maximum_2m_air_temperature',
                            'dewpoint_2m_temperature'])
                   .filterDate('2000-01-01', '2017-12-31')
                   .filterBounds(geometry);

var first_image = era5.first();
var scale_def = first_image.projection().nominalScale();

var era5_over_grids = era5.map(function(image) {
  return image.reduceRegions({
    collection: grids,
    reducer: ee.Reducer.mean(),
    scale: scale_def,
    crs: 'EPSG:4326'
  })});

Export.table.toDrive({
  collection: era5_over_grids,
  description: 'era5_gridded',
  fileFormat: 'CSV'
});

1 Answer 1

1

I had a similar problem, but with a smaller dataset, the result is a table with 140000 rows. When I try the export with the full dataset I wait for about two hours to get the Internal Error message.

This does not happen when I try with a subset of the data (like one year out of 33).
A possible workaround may be to use the integration with another language (R in my case), where I translate the GEE code into R as a function, and iterate it to each year, resulting as a CSV for each year. After that I can concatenate the tables and create another id column.

I don't think it would be the best solution, but could be a workaround while you don't discover the reason for this error.

2
  • I have already tried using the Python API to see if it would solve the problem. I observed the same pattern. If I use fewer polygons, the export works just fine and does not take much time. But as soon as I use the polygons I want, the process never finishes. This happens even when I limit the export to a limited time-period.
    – vpk
    Commented Jul 15, 2020 at 7:28
  • Today I tried this and found a similar behavior, I was able to download some years of data, but got the same errors in other years. The weird thing is that the code was running perfectly two months ago. Maybe posting this issue in the developers group will be a good idea (groups.google.com/g/google-earth-engine-developers). Commented Jul 15, 2020 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.