Earth Engine memory capacity exceeded

Question

I want to generate time-series MODIS images of a small area (10x10km) via Google Earth Engine from 2000 to 2017. I generate the clipped image collection easily and then export each image individually to the cloud. I am using Python API as it is a very long process and should be done in batch. But after a few hours, execution is aborted and I get below message:

Traceback (most recent call last):
  File "Extract_prismV3.py", line 45, in <module>
    date = image.get('system:index').getInfo()
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/computedobject.py", line 95, in getInfo
    return data.getValue({'json': self.serialize()})
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 269, in getValue
    return send_('/value', params)
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 828, in send_
    raise ee_exception.EEException(json_content['error']['message'])
ee.ee_exception.EEException: Earth Engine memory capacity exceeded.

The area is small, and the files are also extracted one by one by two for loops, one for each bounding box and another for each day in the whole period. What might be the reason for such an exception? I don't believe that the code has any problem, because this exception does not occur in a determined way and it comes out to be random and not reproducable at an exact time. I am running Ubuntu 18.04 on a Sony Vaio laptop with 4GB of ram and 64bit AMD E2-2000 processor, and my code is listed below:

# -*- coding: cp1252 -*-
# start: 20:30 8/2/2018

import ee
ee.Initialize()

import time
start = time.time()

MODIS = ee.ImageCollection("MODIS/006/MOD11A1")
Boxes = ee.FeatureCollection("users/shshheydari/thesis/AllBoxes")
boxes = Boxes.sort('boxID')

F = open("MODIS_startDates.txt", 'r')
startDates = F.read().splitlines()
endDate = '2018-01-01'

imageCol = MODIS.select('LST_Day_1km').filterDate('2000-01-01', endDate);
scale = imageCol.first().projection().nominalScale().getInfo();

allBlocks = boxes.map(lambda Bbox:
  ee.Feature(imageCol.map(lambda image:
    image.clip(Bbox).reproject('EPSG:4326', None, scale).multiply(0.02) \
    .set({'boxID': Bbox.get('boxID')})
                          )
             )
                      )
boxes2extract = range(0,168);
for i in boxes2extract:
  if startDates[i] != endDate:
    box = ee.Feature(boxes.toList(boxes.size()).get(i))
    box_id = box.get('boxID').getInfo()
    coords = box.geometry().coordinates().getInfo()
    collection =ee.ImageCollection(allBlocks.toList(allBlocks.size()).get(i)) \
    .filter(ee.Filter.gte('system:index', startDates[i]))
    n = collection.size().getInfo()
    if n > 0:
      for j in range(0, n):
        image = ee.Image(collection.toList(n).get(j))
        fname = "MODIS_"+box_id+"-"+image.get('system:index').getInfo()
        print(fname)
        Task = ee.batch.Export.image.toCloudStorage( 
          image= image,
          description= fname,
          bucket= "modis_clips",
          fileNamePrefix= box_id+"/"+fname,
          region= coords,
          scale= scale
        )
        Task.start()
        while Task.status()['state'] != 'COMPLETED':
          pass

end = time.time()
print 'Run time was',end-start,' seconds'

Answer 1

The error Earth Engine memory capacity exceeded. means that Earth Engine was unable to allocate memory while computing your request. It does not necessarily mean that your script inherently requires too much memory; in that case you would expect to see User memory limit exceeded.

If you are seeing this particular message consistently when you run a script, there might be a problem of interest to the Earth Engine team. If you would like to report it, email earthengine@google.com an explanation, a Get Link link to your script (or, if you can, a simplified version that isn't necessarily scientifically meaningful but does cause the problem), and also share any uploaded assets it requires with that address.