1

Context: I process a set of image using the Google Earth Engine Python API. I want to apply the process on a set of image and save the output (in my case export to GEE Assets).

What I have done: Looping through the image set and export each image one by one. That works well.

Problem: since I would like to apply my process over 60k images, I face the number of task restriction (3000).

Traceback (most recent call last):

ee.ee_exception.EEException: Too many tasks already in the queue (3000). Please wait for some of them to complete.

What I found: the geetools modules. However, by looking at the code, it also seems to iterate over the image collection and exports each image one by one.

Question : Is it possible to export multiple images in one task ? What are the solutions ?

Here the code:

col = ee.ImageCollection("COPERNICUS/S2") \
        .filterDate("2018-01-01", "2018-12-31") \
        .filterBounds(geometry)

# Get names of all image as a python list
image_names = get_name_collection(col).getInfo()
total = len(image_names)

# Create new image collection if don't existe
folder = "users/ab43536/masks_4_methods"
if folder not in [elt["id"] for elt in ee.data.getList({"id": "users/ab43536"})]:
    ee.data.createAsset({'type': "ImageCollection"}, folder)

# Apply process to all images
for i, name in enumerate(image_names):
    print("{:4d}/{} = {:05.2f}%   Image {}".format(i, total, i / total * 100, name))
    # The long process
    mask = computeCloudMasking(name)

    # Export image one by one
    export_image(mask, folder, getGeometryImage(ee.Image(name)),
                 name.split('/')[-1], num=i, total=total)

Where export_image is:

def export_image(image, asset_id="users/ab43536/", roi=None, name=None, num=None, total=None):
    """ Export one image to GEE Asset
    Arguments
        :param image: image to export
        :param roi=None:  specify the roi, default compute from image dimension
        :param name=None: name of the image
    """
    if roi == None: roi = getGeometryImage(image)
    if name == None: name = image.id().getInfo()
    description = "Default export"
    if num != None and total != None:
        description = "Image {} on {} equal {:05.2f} pourcent".format(num, total, num / total * 100)
    # print(description)
    assetId = asset_id + name

    # Create a task : export the result as image asset
    task = ee.batch.Export.image.toAsset(image=image.clip(roi),
                                         description=description,
                                         assetId=assetId,
                                         scale=30,
                                         region=roi.coordinates().getInfo(),
                                         )
    task.start()
  • 1
    Hi Alexander, the developer of geetools here. You can ask your question here if you want. It has a way to export an ImageCollection as an ImageCollection asset – Rodrigo E. Principe Jul 5 at 11:03
  • Hi @RodrigoE.Principe. Thank you for reply. Thanks to your work, I founded how to create an ImageCollection using: (ee.data.createAsset({'type':"ImageCollection"}, "users/.../..."). – Alexandre B. Jul 5 at 14:36
  • 1
    You're right, geetools is not handling this in a proper way, I'll fix that! Thanks! – Rodrigo E. Principe Jul 5 at 15:29
  • If you want it to get out of on hold you have to include some code. What you have tried, what didn't work, errors.. – Rodrigo E. Principe Jul 6 at 12:39
  • @RodrigoE.Principe Why using your code I get ee.batch.Export.image.toAsset(...) => "can't overwrite"? – Giox Oct 4 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.