Exporting Large Number of Landsat Images from Google Earth Engine

Question

I am interested in downloading tiff images of cloud-free landsat data from around 100,000 areas of interest in Google Earth Engine. I have written the export script but the process of exporting takes a long time. I am doing this on the web code-editor. I want to know if there is a faster way of doing this. Will offline downloading using python ee package work faster?

I am interested in downloading from GEE as it contains processed clean data with clips and other such functionality and hence am not looking to use directly available data perhaps on USGS site which would also require me to use Qgis and do much of the filtering and pre-processing on my own...

Answer 1

The first thing you have to know is that Google Earth Engine is a platform where you can process and analyse raster data (satellite images, and more), and then, download the processed data if you need. I think it is its main purpose. Sure, there may be some processes that you cannot do on it, and in that case you could download the unprocessed images.

Use the Python API. First install it: https://developers.google.com/earth-engine/python_install. If you are using Windows, you can do the 'Minimal Installation' using Anaconda (https://www.anaconda.com/download).

When you have the Python API running. Install geetools (https://github.com/gee-community/gee_tools). In that package there is a function called col2drive.

from geetools import batch
import ee

col = ee.ImageCollection('XXX')
batch.ImageCollection.toDrive(col, 'FOLDER', scale=30) # run help(batch.ImageCollection.toDrive) to see the help

In the develop branch may be (and will be) more and better features. I am working in a toLocal function, to download the collection directly to your local machine.

IMPORTANT: Reaching quota limits is your responsibility

Answer 2

using GEE might not be a great idea. Since for a single account they also cap the number of tasks So if you just keep queuing Export tasks, you'll get

ee.ee_exception.EEException: Too many tasks already in the queue (3001). Please wait for some of them to complete.