2

I want to generate time-series MODIS images of a small area (10x10km) via Google Earth Engine from 2000 to 2017. I generate the clipped image collection easily and then export each image individually to the cloud. I am using Python API as it is a very long process and should be done in batch. But after a few hours, execution is aborted and I get below message:

Traceback (most recent call last):
  File "Extract_prismV3.py", line 45, in <module>
    date = image.get('system:index').getInfo()
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/computedobject.py", line 95, in getInfo
    return data.getValue({'json': self.serialize()})
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 269, in getValue
    return send_('/value', params)
  File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 828, in send_
    raise ee_exception.EEException(json_content['error']['message'])
ee.ee_exception.EEException: Earth Engine memory capacity exceeded.

The area is small, and the files are also extracted one by one by two for loops, one for each bounding box and another for each day in the whole period. What might be the reason for such an exception? I don't believe that the code has any problem, because this exception does not occur in a determined way and it comes out to be random and not reproducable at an exact time. I am running Ubuntu 18.04 on a Sony Vaio laptop with 4GB of ram and 64bit AMD E2-2000 processor, and my code is listed below:

# -*- coding: cp1252 -*-
# start: 20:30 8/2/2018

import ee
ee.Initialize()

import time
start = time.time()

MODIS = ee.ImageCollection("MODIS/006/MOD11A1")
Boxes = ee.FeatureCollection("users/shshheydari/thesis/AllBoxes")
boxes = Boxes.sort('boxID')

F = open("MODIS_startDates.txt", 'r')
startDates = F.read().splitlines()
endDate = '2018-01-01'

imageCol = MODIS.select('LST_Day_1km').filterDate('2000-01-01', endDate);
scale = imageCol.first().projection().nominalScale().getInfo();

allBlocks = boxes.map(lambda Bbox:
  ee.Feature(imageCol.map(lambda image:
    image.clip(Bbox).reproject('EPSG:4326', None, scale).multiply(0.02) \
    .set({'boxID': Bbox.get('boxID')})
                          )
             )
                      )
boxes2extract = range(0,168);
for i in boxes2extract:
  if startDates[i] != endDate:
    box = ee.Feature(boxes.toList(boxes.size()).get(i))
    box_id = box.get('boxID').getInfo()
    coords = box.geometry().coordinates().getInfo()
    collection =ee.ImageCollection(allBlocks.toList(allBlocks.size()).get(i)) \
    .filter(ee.Filter.gte('system:index', startDates[i]))
    n = collection.size().getInfo()
    if n > 0:
      for j in range(0, n):
        image = ee.Image(collection.toList(n).get(j))
        fname = "MODIS_"+box_id+"-"+image.get('system:index').getInfo()
        print(fname)
        Task = ee.batch.Export.image.toCloudStorage( 
          image= image,
          description= fname,
          bucket= "modis_clips",
          fileNamePrefix= box_id+"/"+fname,
          region= coords,
          scale= scale
        )
        Task.start()
        while Task.status()['state'] != 'COMPLETED':
          pass

end = time.time()
print 'Run time was',end-start,' seconds'
  • Please Edit the question to contain details about the computer hardware, OS platform, version and application bit depth. An indication of memory use over time from the OS would also be useful. – Vince Aug 11 '18 at 11:28
  • The question is not clear, you should post some code, at least a minimum code that reproduces the issue. Creating a time-series from a MODIS collection in a 10x10 km area should not give you that error. This error has nothing to do with your system OS or memory. It means that you are trying to use more Earth Engine memory than is assigned to your user. – Rodrigo E. Principe Aug 11 '18 at 15:50
  • Thanks for your attention and sorry for my incomplete first question. I was out of town in the past days but now updated it and hope I can get some help now. – Shahriar49 Aug 14 '18 at 10:59
3

The error Earth Engine memory capacity exceeded. means that Earth Engine was unable to allocate memory while computing your request. It does not necessarily mean that your script inherently requires too much memory; in that case you would expect to see User memory limit exceeded.

If you are seeing this particular message consistently when you run a script, there might be a problem of interest to the Earth Engine team. If you would like to report it, email earthengine@google.com an explanation, a Get Link link to your script (or, if you can, a simplified version that isn't necessarily scientifically meaningful but does cause the problem), and also share any uploaded assets it requires with that address.

  • Thanks Kevin for reply. I am trying and will contact them if the problem persists. – Shahriar49 Aug 16 '18 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.