I have indexed 5 Sentinel-2 images and I am requesting the data (10 bands and the 5 images) for each one of 180,000 polygons included in a shapefile.

Each request needs 8 seconds for the related data to be retrieved. I use a 32GB VM with 8 cores.

Is there any alternative and more efficient way to make the request other than the code below ?

I tried also by loading firstly all the data and then slicing based on the polygon's bounds.

Finally, do you think multiprocessing will help?

shape_file = '/data2/cyprus/cyprus_parcels_2019_36SWD_3857.shp'
ds = fiona.open(shape_file)
crs = geometry.CRS(ds.crs_wkt)
for f in ds:
 feature_geom = f['geometry']
 geom = geometry.Geometry(geom=feature_geom,crs=crs)
 bounds = shape(feature_geom).bounds
 if 'MULTIPOLYGON' in geom.wkt:
 product = "s2a_sen2cor_granule"
 query = {
    'time': ('2019-01-01', '2019-12-31'),
    'product': product
 data = 

I think you need to design your analysis to fit into memory and optimise so that you're only loading data from the Sentinel-5 data once.

Here's a suggestion:

  1. Load all your polygons
  2. Use the find_datasets command in the ODC to identify the extents of each dataset
  3. Do an intersection with polygons and datasets, to group the polygons by dataset
  4. Load one dataset, do the extraction using the polygons that match that dataset and save the results, delete the in-memory dataset
  5. Load the next dataset and repeat 4.

If that still runs out of memory, you'll need to partition into smaller parts, perhaps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.