Skip to main content
Post Reopened by TomazicM, PolyGeo
deleted 16 characters in body
Source Link
PolyGeo
  • 65.4k
  • 29
  • 114
  • 345

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

How would you recommend creatingcan I create an efficient server-side loop to export an entire FeatureCollection?

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

How would you recommend creating an efficient server-side loop to export an entire FeatureCollection?

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

How can I create an efficient server-side loop to export an entire FeatureCollection?

deleted 858 characters in body
Source Link

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

I have tried creating a function and using .map to apply it to the whole FeatureCollection, but that fails to create unique names! Furthermore, I can't make the .map solution work without the getInfo() in the middle, which means there wouldn't be that much gain anyway.

Opportunities for improvement, but which I can't seem to get to work:

  1. Actually BATCH process the FeatureCollection: instead of creating one Task for each Feature, it seems like it should be possible to create one Task for the entire FeatureCollection (otherwise, the batch API is very poorly named!)

  2. Create a loop on the server side: instead of creating the loop on the client side, there's no reason why I couldn't write an ee.ForEach, but I can't find anything like this structure in the documentation.

  3. If I preprocess the entire FeatureCollection with a single getInfo(), then reshape it into an easy-to-extract ee.Dictionary, for some reason I still can't figure out the syntax for reading it on the server side. "myEEdict.get(feature.id())" returns the error "TypeError: Object of type 'ComputedObject' is not JSON serializable".

How would you recommend creating an efficient server-side loop to export an entire FeatureCollection?

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

I have tried creating a function and using .map to apply it to the whole FeatureCollection, but that fails to create unique names! Furthermore, I can't make the .map solution work without the getInfo() in the middle, which means there wouldn't be that much gain anyway.

Opportunities for improvement, but which I can't seem to get to work:

  1. Actually BATCH process the FeatureCollection: instead of creating one Task for each Feature, it seems like it should be possible to create one Task for the entire FeatureCollection (otherwise, the batch API is very poorly named!)

  2. Create a loop on the server side: instead of creating the loop on the client side, there's no reason why I couldn't write an ee.ForEach, but I can't find anything like this structure in the documentation.

  3. If I preprocess the entire FeatureCollection with a single getInfo(), then reshape it into an easy-to-extract ee.Dictionary, for some reason I still can't figure out the syntax for reading it on the server side. "myEEdict.get(feature.id())" returns the error "TypeError: Object of type 'ComputedObject' is not JSON serializable".

How would you recommend creating an efficient server-side loop to export an entire FeatureCollection?

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

How would you recommend creating an efficient server-side loop to export an entire FeatureCollection?

Post Closed as "Needs more focus" by PolyGeo
Source Link

Server-side loop export

Given a FeatureCollection that contains multiple Features (each consisting of polygons), I would like to use the Python API to loop through them all, and batch export them to Google Cloud Storage.

However, I can't figure out how to make all the code server-side, and hence my code runs extremely slowly (using the major no-no of getInfo inside a loop).

for i in range(0, 1000):
  region = ee.Feature(myFeatCol.get(i)).geometry()

  task = ee.batch.Export.image.toCloudStorage(
    image = naip.clip(region),
    description = 'uniqueName'+str(i),
    fileNamePrefix = 'uniqueName'+str(i),
    bucket = 'myGoogleBucketName',
    scale = 1,
    region = region.getInfo()['coordinates'])

  task.start()

The objective here is to end up with 1000 images in the bucket. It works, but is painstakingly slow! Several-minutes-per-image slow.

I have tried creating a function and using .map to apply it to the whole FeatureCollection, but that fails to create unique names! Furthermore, I can't make the .map solution work without the getInfo() in the middle, which means there wouldn't be that much gain anyway.

Opportunities for improvement, but which I can't seem to get to work:

  1. Actually BATCH process the FeatureCollection: instead of creating one Task for each Feature, it seems like it should be possible to create one Task for the entire FeatureCollection (otherwise, the batch API is very poorly named!)

  2. Create a loop on the server side: instead of creating the loop on the client side, there's no reason why I couldn't write an ee.ForEach, but I can't find anything like this structure in the documentation.

  3. If I preprocess the entire FeatureCollection with a single getInfo(), then reshape it into an easy-to-extract ee.Dictionary, for some reason I still can't figure out the syntax for reading it on the server side. "myEEdict.get(feature.id())" returns the error "TypeError: Object of type 'ComputedObject' is not JSON serializable".

How would you recommend creating an efficient server-side loop to export an entire FeatureCollection?