I want to generate time-series MODIS images of a small area (10x10km) via Google Earth Engine from 2000 to 2017. I generate the clipped image collection easily and then export each image individually to the cloud. I am using Python API as it is a very long process and should be done in batch. But after a few hours, execution is aborted and I get below message:
Traceback (most recent call last):
File "Extract_prismV3.py", line 45, in <module>
date = image.get('system:index').getInfo()
File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/computedobject.py", line 95, in getInfo
return data.getValue({'json': self.serialize()})
File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 269, in getValue
return send_('/value', params)
File "/home/shahriar49/.local/lib/python2.7/site-packages/ee/data.py", line 828, in send_
raise ee_exception.EEException(json_content['error']['message'])
ee.ee_exception.EEException: Earth Engine memory capacity exceeded.
The area is small, and the files are also extracted one by one by two for
loops, one for each bounding box and another for each day in the whole period. What might be the reason for such an exception? I don't believe that the code has any problem, because this exception does not occur in a determined way and it comes out to be random and not reproducable at an exact time. I am running Ubuntu 18.04 on a Sony Vaio laptop with 4GB of ram and 64bit AMD E2-2000 processor, and my code is listed below:
# -*- coding: cp1252 -*-
# start: 20:30 8/2/2018
import ee
ee.Initialize()
import time
start = time.time()
MODIS = ee.ImageCollection("MODIS/006/MOD11A1")
Boxes = ee.FeatureCollection("users/shshheydari/thesis/AllBoxes")
boxes = Boxes.sort('boxID')
F = open("MODIS_startDates.txt", 'r')
startDates = F.read().splitlines()
endDate = '2018-01-01'
imageCol = MODIS.select('LST_Day_1km').filterDate('2000-01-01', endDate);
scale = imageCol.first().projection().nominalScale().getInfo();
allBlocks = boxes.map(lambda Bbox:
ee.Feature(imageCol.map(lambda image:
image.clip(Bbox).reproject('EPSG:4326', None, scale).multiply(0.02) \
.set({'boxID': Bbox.get('boxID')})
)
)
)
boxes2extract = range(0,168);
for i in boxes2extract:
if startDates[i] != endDate:
box = ee.Feature(boxes.toList(boxes.size()).get(i))
box_id = box.get('boxID').getInfo()
coords = box.geometry().coordinates().getInfo()
collection =ee.ImageCollection(allBlocks.toList(allBlocks.size()).get(i)) \
.filter(ee.Filter.gte('system:index', startDates[i]))
n = collection.size().getInfo()
if n > 0:
for j in range(0, n):
image = ee.Image(collection.toList(n).get(j))
fname = "MODIS_"+box_id+"-"+image.get('system:index').getInfo()
print(fname)
Task = ee.batch.Export.image.toCloudStorage(
image= image,
description= fname,
bucket= "modis_clips",
fileNamePrefix= box_id+"/"+fname,
region= coords,
scale= scale
)
Task.start()
while Task.status()['state'] != 'COMPLETED':
pass
end = time.time()
print 'Run time was',end-start,' seconds'