0

I am attempting to export an ImageCollection from Google Earth Engine to Cloud Storage using python API.
I work in Atom text editor using Hydrogen, which relies on Jupyter. My workstation runs Ubuntu 20.04.

The code below does not return any error, but the 212 images in collection are not exported either. Nonetheless, I can copy/remove objects (e.g. a_test_file) manually to/from the bucket using gsutil. This confirms that I have WRITE permission to the bucket (actually I have OWNER permission).
I have tried to go through the native export tool of Earth Engine and those provided by geetools. None of the them led to the expected outcome.

Is there any specific setting/configuration necessary to export from GEE to Cloud Storage which I should take care of beforehand? I have checked the documentation extensively, but I may have missed something, of course. Any suggestion is welcome.

import ee
import geetools

# ee.Authenticate()
ee.Initialize()

aoi_coo = [[44.1475, -0.8456],
           [44.73383, 4.08383],
           [44.62833, 4.56666],
           [43.88367, 5.16483],
           [42.96, 2.53],
           [43.6397, -0.435]]

aoi = ee.Geometry.Polygon(coords=aoi_coo).buffer(100)

collection = ee.ImageCollection("COPERNICUS/S1_GRD")\
            .filterDate('2018-01-01', '2019-01-01')\
            .filterBounds(aoi)

vv_desc = collection.filter(ee.Filter.listContains('transmitterReceiverPolarisation', 'VV'))\
    .filter(ee.Filter.eq('instrumentMode', 'IW'))\
    .filter(ee.Filter.eq('orbitProperties_pass', 'DESCENDING'))

vv_desc.size().getInfo()

# Export parameters
out_bucket = 'my_bucket'
name_pattern = 'system:id'
scale = 10

# Test copy/remove object to/from bucket using gsutil tools
!touch a_test_file
!gsutil cp a_test_file gs://my_bucket
!gsutil ls -l gs://my_bucket/
!gsutil rm gs://my_bucket/a_test_file
### ALL GOOD!

### Native ee approach
imagesList = vv_desc.toList(vv_desc.size())
for i in range(0, imagesList.size().getInfo()):
    image= ee.Image(imagesList.get(i))
    #Process the export for you image into the bucket in GCS
    task = ee.batch.Export.image.toCloudStorage(**{
        'image':image,
        'region':aoi_coo,
        'description':'Sent1 image',
        'fileNamePrefix':image.get(name_pattern).getInfo() + '.tif',
        'scale':scale,
        'bucket':out_bucket,
        'fileFormat':'GeoTIFF'})
    task.start()

!gsutil ls gs://my_bucket/

### geetools approach
tasks = geetools.batch.imagecollection.toCloudStorage(
            collection=vv_desc,
            bucket=out_bucket,
            region=aoi,
            namePattern=name_pattern,
            scale=scale,
            dataType='float',
            verbose=True
            )

!gsutil ls gs://my_bucket/

1
  • Running my code on other machines/accounts and letting me know the outcome would be helpful too. I'm pretty sure there is nothing wrong with the code itself, so the issue must lie elsewhere in the system
    – IvanP
    Oct 22 '20 at 7:33
0

I can confirm that both solutions posted in my question (i.e. geetools and native export tool from Google Earth Engine) do actually export the image collection to my_bucket in Google Cloud. Worth mentioning that the login to GCloud must be performed in advance: gcloud auth login is the command to login from terminal. Despite this, I keep observing anomalies in the platform, which may be actually limited to the python API, because the code does not always export the image collection as expected. I could not find a pattern between the cases when image collections are successfully exported to Google Cloud, and cases when they are not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.