Creating large covariate stacks with Google Earth Engine

Question

I'm trialling using Earth Engine for a machine learning task (predicting pH) based on a large number (~300) of satellite covariates and around 100k training points. Earth Engine is attractive due to the ease of covariate preparation, and I'm following the basic process outlined in Google's example colab notebooks to:

• Create an image with a band for each covariate
• Use ee.Image.sampleRegions() to extract the values of the bands per training point as a training set
• Train a model using Tensorflow on this training set
• Follow the EEification process and deploy model to AI Platform, then perform inference in Earth Engine by loading the model with ee.Model.fromAiPlatformPredictor().

I've proved out this route end-to-end with a small covariate stack, but have now hit a snag when creating a larger covariate stack. At about 50 bands in the stack attempting an export of the training points triggers the error:

Error: Image.reduceRegions: Computed value is too large.

I've tried bumping the tileScale parameter on sampleRegions to 16 but this still fails (but takes much longer - around 3hrs instead of 3mins with tileScale set to the default 1).

Can anyone give any pointers to the recommended route here? I'd like to avoid exporting separate stacks / collections of training points and feel this size of operation should be possible based on what I've read, but I haven't been able to find much online regarding large (100+ band) stacks on Earth Engine. Very open to other approaches if this is working against the way Earth Engine is intended to be used.

This example code demonstrates the issue, and is available to run directly on Earth Engine here: https://code.earthengine.google.com/7b72b2909940c395fab86fb111383962

var admin1 = ee.FeatureCollection("FAO/GAUL_SIMPLIFIED_500m/2015/level1");
var ghana = admin1.filter(ee.Filter.eq('ADM0_NAME', 'Ghana'));

/* 
In reality these training points come from a separate ee.FeatureCollection containing
the ground truth data used to train the model which created this image, but here we
generate them by sampling for 100k points from Ghana for simplicity.
*/
var training_points = ee.Image('projects/isdasoil/soil_data/ph')
  .select('mean_0_20')
  .sample(
    ghana, // geometry
    null, // scale
    null, // projection
    null, // factor
    1e5, // numPixels
    0, // seed
    true, // dropNulls
    1, // tileScale
    true // include geometry - needed for sampleRegions
  );
  
print("Training points generated: ", training_points.size());

// Uncomment to visualise
// Map.centerObject(ghana, 8);
// Map.addLayer(training_points);

var dem_30m = ee.Image('projects/isdasoil/covariates/dem_30m');
var dem_100m = ee.Image('projects/isdasoil/covariates/dem_100m');
var landcover_100m = ee.Image('projects/isdasoil/covariates/landcover_100m');
var landcover_30m = ee.Image('projects/isdasoil/covariates/landcover_30m');
var landsat7_2000 = ee.Image('projects/isdasoil/covariates/landsat7_2000');
var landsat8_2015 = ee.Image('projects/isdasoil/covariates/landsat8_2015');
var landsat8_2018 = ee.Image('projects/isdasoil/covariates/landsat8_2018');
var sentinal2_l2a_s1 = ee.Image('projects/isdasoil/covariates/sentinel2_l2a_s1');
var sentinel2_l2a_s2 = ee.Image('projects/isdasoil/covariates/sentinel2_l2a_s2');
var surface_water = ee.Image('projects/isdasoil/covariates/surface_water');

var stack = dem_30m
  .addBands(dem_100m)
  .addBands(landcover_100m)
  .addBands(landcover_30m)
  .addBands(landsat7_2000)
  .addBands(landsat8_2015)
  .addBands(landsat8_2018)
  .addBands(sentinal2_l2a_s1)
  .addBands(sentinel2_l2a_s2)
  .addBands(surface_water);

var stacked_training_points = stack.sampleRegions(
  training_points,
  null, // all properties
  30, // scale
  null, // projection
  16, // tilescale
  true // include geometry - needed to export as Asset
);

Export.table.toAsset(
  stacked_training_points,
  "test_export_of_stacked_training_points"
);

// Now need to click RUN in Tasks panel
// Even with tileScale set to 16 fails with 
// Error: Image.reduceRegions: Computed value is too large.

Answer 1

You should be able to map over your points and sample them one by one.

var stacked_training_points = training_points.map(
  function (training_point) {
    return stack.sampleRegions({
      collection: ee.FeatureCollection([training_point]),  
      scale: 30,
      geometries: true
    }).first()
  }, true) // Set dropNulls to true - exclude points where stack is masked out.

https://code.earthengine.google.com/b26ed5a2b1b78f17cdfbb28626a47426