1

I was trying to merge two Sentinel-2 image collections (one based on date filtering and one based on image IDs), then pull in the matching Sentinel-2: Cloud Probability dataset to apply cloud masking to the merged collection.

However, it appears that performing a merge operation modifies the image "system:index" values (which are needed for the join with the Cloud probability dataset). The merge adds "1_" and "2_" prefixes to the "system:index" of the images in the merged collection. This stops any subsequent joining with another dataset from working.

The following is a simplified code of what I am trying to achieve. The problem is that the join with the Sentinel-2: Cloud Probability returns an empty set of images due to the prefixes added by the merge operation. If I simply join to one of the un-merged image collections, such as col1, the join works fine.

The "system:id" of the images are not modified by the merge operation and so I tried joining on the "system:id" instead of "system:index" but I received an error indicating:

FeatureCollection (Error) Cannot filter on 'system:id'. Apply the filter to property 'system:index' instead.

Is there a way to merge image collections without the "system:index" properties being modified?

var testArea =  ee.Geometry.BBox(146.5, -18.3, 146.55, -18.2);
          
// Create two image collections. One based on image IDs, the
// second based on date filtering.
var IMAGE_IDS =  ["COPERNICUS/S2/20170104T002702_20170104T002921_T55KDV"];

var manualImages = IMAGE_IDS.map(function(id) {
    return ee.Image(id);
  });
  
var col1 = ee.ImageCollection(manualImages);
    
var col2 = ee.ImageCollection('COPERNICUS/S2')
    .filterBounds(testArea)
    .filterDate('2018-01-01', '2018-01-05');

// Verify the system:index prior to merge
var col1IDs = col1.aggregate_array('system:index');
print(col1IDs);
// ["20170104T002702_20170104T002921_T55KDV"]
  
var col2IDs = col2.aggregate_array('system:index');
print(col2IDs);
// ["20180104T002659_20180104T003132_T55KDV"]

// Merge the two collections into one collection
var mergedCollection = col1.merge(col2);

var mergedIDs = mergedCollection.aggregate_array('system:index');
print(mergedIDs);
// List (2 elements)
// 0: 1_20170104T002702_20170104T002921_T55KDV
// 1: 2_20180104T002659_20180104T003132_T55KDV
// The system:index has been modified to include 1_ and 2_ as prefixes

// Attempt to bind the S2_CLOUD_PROBABILITY dataset to these images
// Extract from https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S2_CLOUD_PROBABILITY
var s2Clouds = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')
  .filterBounds(testArea);

// Join S2 with cloud probability dataset to add cloud mask.
var s2SrWithCloudMask = ee.Join.saveFirst('cloud_mask').apply({
  primary: mergedCollection,
  secondary: s2Clouds,
  condition:
      ee.Filter.equals({leftField: 'system:index', rightField: 'system:index'})
});

print(s2SrWithCloudMask)
// ImageCollection (0 elements)
// No match due to 1_ and 2_ prefixes from merge.```

1 Answer 1

1

Is there a way to merge image collections without the "system:index" properties being modified?

No, it is not possible. This is because of two fundamental properties of Earth Engine collections:

  1. system:index values are unique within a collection (no two features have the same ID).
  2. Collections are processed in a “streaming” fashion: processing each feature does not require knowledge of previous features.

If a merge operation did not modify the IDs then to avoid violating property (1), it would need to remember previous IDs and violate property (2).


What you can do is join on a property you create that isn't system:index, which will thus not be modified by the merge.

function preserveId(image) {
  return image.set('original_id', image.get('system:index'));
}

...

var col1 = ee.ImageCollection(manualImages).map(preserveId);
    
var col2 = ee.ImageCollection('COPERNICUS/S2')
    .filterBounds(testArea)
    .filterDate('2018-01-01', '2018-01-05')
    .map(preserveId);

var mergedCollection = col1.merge(col2);

var s2Clouds = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')
  .filterBounds(testArea)
  .map(preserveId);

var s2SrWithCloudMask = ee.Join.saveFirst('cloud_mask').apply({
  primary: mergedCollection,
  secondary: s2Clouds,
  condition:
      ee.Filter.equals({leftField: 'original_id', rightField: 'original_id'})
});

print(s2SrWithCloudMask);

https://code.earthengine.google.com/d6cfe50ba052d878cf71c00ad0ffa5db

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.