2

I am creating training and test data for a Random Forest model using Google Earth Engine (Python API).

I have used stratifiedSample() to create a Feature Collection of sample points within my training polygons. I would like to use randomColumn() to assign a pseudo-random float to each feature so I can then split them into training and test sets using a threshold.

How do I ensure the random numbers are assigned per strata, in this case per the column class?

I see that ee.Reducer has a group method but this is not applicable for Feature Collections.

My current code, in which I am unsure if the strata were respected when adding the randomColumn:

## make a feature collection of training polygons
fc = ee.FeatureCollection(polygons)

## create an empty object to hold stratified points
classes = ee.Image().byte().paint(polygons, "class").rename("class")

## stratify sample points per feature
stratified = classes.addBands(ee.Image.pixelLonLat()).stratifiedSample(
      numPoints = 1000,
      classBand = 'class',
      projection = 'EPSG:4326',
      scale = 10,
      region = fc,
      geometries = True
    )

## add a random number to each feature   
stratified = stratified.randomColumn()

## sample image using stratified points ('random' is the default column name given by randomColumn)
sample = img.select(train_bands).sampleRegions(collection=stratified, properties=['class', 'random'], scale=10)

## define fraction for training (remainder is for testing)
split = 0.7

## divide into training and testing sets based on the split
training = sample.filter(ee.Filter.lt('random', split))
validation = sample.filter(ee.Filter.gte('random', split))

1 Answer 1

3

The random numbers from randomColumn are uniformly distributed. That means just taking 30% of the whole collection will take 30% of each strata, to within a small statistical margin. You can tell how many in each with a reduceColumns and a frequencyHistogram on the class property, after the filter.

If you really think you need an exact amount of each class (seems unlikely) then you should run stratified sample twice, once for training (numPoints: 700) and once for validation (numPoints: 300).

To guarantee that there are no validation points within a certain distance of the training points (or on top of them), you can mask the inputs with a (potentially buffered) version of the training points before taking the validation sample.

Also:

  1. you don't really need to add a pixelLonLat band AND run it with {geometries: true}. The geometries already have the lon/lat in them.
  2. You shouldn't run stratifiedSample and then sampleRegions on the outputs; just add the training bands to the classes and run stratifiedSample on that whole thing.
1
  • 1
    Thank you, Noel. After a rethink, I realised that indeed the strata will be sampled uniformly and I was actually about to take my question down until I saw your answer. Thank you for the extra tips, which I have now implemented.
    – Matt
    Sep 1, 2021 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.