1

Context: I process a set of image using the Google Earth Engine Python API. I want to apply the process on a set of image and save the output (in my case export to GEE Assets).

What I have done: Looping through the image set and export each image one by one. That works well.

Problem: since I would like to apply my process over 60k images, I face the number of task restriction (3000).

Traceback (most recent call last):

ee.ee_exception.EEException: Too many tasks already in the queue (3000). Please wait for some of them to complete.

What I found: the geetools modules. However, by looking at the code, it also seems to iterate over the image collection and exports each image one by one.

Question : Is it possible to export multiple images in one task ? What are the solutions ?

Here the code:

col = ee.ImageCollection("COPERNICUS/S2") \
        .filterDate("2018-01-01", "2018-12-31") \
        .filterBounds(geometry)

# Get names of all image as a python list
image_names = get_name_collection(col).getInfo()
total = len(image_names)

# Create new image collection if don't existe
folder = "users/ab43536/masks_4_methods"
if folder not in [elt["id"] for elt in ee.data.getList({"id": "users/ab43536"})]:
    ee.data.createAsset({'type': "ImageCollection"}, folder)

# Apply process to all images
for i, name in enumerate(image_names):
    print("{:4d}/{} = {:05.2f}%   Image {}".format(i, total, i / total * 100, name))
    # The long process
    mask = computeCloudMasking(name)

    # Export image one by one
    export_image(mask, folder, getGeometryImage(ee.Image(name)),
                 name.split('/')[-1], num=i, total=total)

Where export_image is:

def export_image(image, asset_id="users/ab43536/", roi=None, name=None, num=None, total=None):
    """ Export one image to GEE Asset
    Arguments
        :param image: image to export
        :param roi=None:  specify the roi, default compute from image dimension
        :param name=None: name of the image
    """
    if roi == None: roi = getGeometryImage(image)
    if name == None: name = image.id().getInfo()
    description = "Default export"
    if num != None and total != None:
        description = "Image {} on {} equal {:05.2f} pourcent".format(num, total, num / total * 100)
    # print(description)
    assetId = asset_id + name

    # Create a task : export the result as image asset
    task = ee.batch.Export.image.toAsset(image=image.clip(roi),
                                         description=description,
                                         assetId=assetId,
                                         scale=30,
                                         region=roi.coordinates().getInfo(),
                                         )
    task.start()
5
  • 1
    Hi Alexander, the developer of geetools here. You can ask your question here if you want. It has a way to export an ImageCollection as an ImageCollection asset Jul 5 '19 at 11:03
  • Hi @RodrigoE.Principe. Thank you for reply. Thanks to your work, I founded how to create an ImageCollection using: (ee.data.createAsset({'type':"ImageCollection"}, "users/.../..."). Jul 5 '19 at 14:36
  • 1
    You're right, geetools is not handling this in a proper way, I'll fix that! Thanks! Jul 5 '19 at 15:29
  • If you want it to get out of on hold you have to include some code. What you have tried, what didn't work, errors.. Jul 6 '19 at 12:39
  • @RodrigoE.Principe Why using your code I get ee.batch.Export.image.toAsset(...) => "can't overwrite"?
    – Giox
    Oct 4 '19 at 12:24
0

Processing that amount of images in a stack would be too big, but you can export multiple images in one task by stacking the images, then exporting the stack.

Here's my code for the Java version, if you know how to translate:

// Stack collection to output image

var tsStack = function(image, list) {
    return ee.Image(list).addBands(image.select([bands]));
};
var stack = ee.Image(newcol.iterate(tsStack,ee.Image(0.0).toDouble())).slice(1)

Export.image.toDrive({image:stack,description:name+'_stack',folder:'yourfolder', region:roi, scale:scale, crs:crs})

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.