I'd like to export time-series data for many different lakes across the U.S. However, I run into memory issues when trying to start the process.

To handle my processes, I'm using the Python API and ee.batch.table.toDrive. I'm using a loop, like so:

nhd_list = nhd.toList(nhd.size())
for i in range(0, ee.Number(nhd_list.size()).getInfo()):
    feat = ee.Feature(nhd_list.get(i))
    geom = feat.geometry()
    name = str(ee.String(feat.get("NHDPlID")).getInfo())
    #A series of processing steps for export
    #By this step, pixels have been filtered and reductions to 
    #get medians have been performed. 
    task = ee.batch.Export.table.toDrive(
        **{'fileFormat': 'CSV', 
           'driveFolder': 'Test', 
           'fileNamePrefix': name})

Unsurprisingly, when I try to run this, I receive an error that the user memory limit has exceeded. However, it works when I limit the tasks to 3000 at a time.

My plan is to thus keep a loop going and wait until the amount of tasks falls below some number, after which the next task starts.

Is there a way to view the amount of running tasks through the Python API? Or am I looking at this the wrong way? Is there a more efficient way of handling this?


The problem is not what you think:

  • Batch tasks go into a queue and do not consume any resources until they start running.
  • Memory limits are independent, not shared across all tasks.

The problem you are having is not about your batch tasks, but what you're doing to set them up.

nhd_list = nhd.toList(nhd.size())

This line means that any computation that depends on nhd_list brings the entire contents of nhd into memory. Then your two .getInfo() calls are such computations, which are run immediately without ever involving the batch task system.

What you should do, instead of converting your entire data set into a list, convert only its IDs, then download those IDs with getInfo() and use them to filter your collection.

# Make a FeatureCollection with no properties except the id / system:index property
nhd_id_only = nhd.select(propertySelectors=[], retainGeometry=False)

# Download the list of IDs
nhd_id_list = (nhd_id_only
    .map(lambda feature: ee.Feature(feature).id())

# Loop over IDs
for feature_id in nhd_id_list:
    # Find the feature again by its ID in the collection.
    feat = nhd.filter(ee.Filter.eq('system:index', feature_id)).first()

    # now do your analysis on `feat`

(Disclaimer: I haven't actually run this code in a Python environment, so there might be some small mistakes.)

Note also that I have replaced nhd.toList(nhd.size()) with nhd.toList(1_000_000). This is more efficient, because both toList and size have to read the entire collection; the number is merely a limit on the number of items collected in the list. So you should just set it to a value known to be bigger than your collection.

  • Thanks for the clarification. Is there a limit on the amount of concurrent tasks that can be running? – Joshua Mincer Aug 7 '20 at 19:17
  • 1
    @JoshuaMincer The number of concurrently running batch tasks is determined by how many other users are also running batch tasks and are therefore sharing the same servers your tasks are using. But you can request up to 3000 tasks and they will sit in the queue and run as servers become available. If your dataset is larger than that, you'll need to either wait for some tasks to finish, or export more data in fewer tasks/files. – Kevin Reid Aug 7 '20 at 19:53
  • That makes sense. So, if I request up to 10,000 tasks, will I be thrown an error, or will these just wait to be processed on the GEE server end? – Joshua Mincer Aug 7 '20 at 19:55
  • 1
    @JoshuaMincer You will be unable to submit more tasks; you'll see an error immediately. In fact, that might have been what already happened, since I see you mentioned a count of 3000. My answer (while still a good idea for efficiency) was based on assuming you in fact saw a memory error. – Kevin Reid Aug 7 '20 at 21:33
  • 1
    @JoshuaMincer Yes, ee.data.getTaskList / ee.data.listOperations. I haven't used them myself to give any detailed advice. – Kevin Reid Aug 7 '20 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.