1

I'm estimating forest biomass using Random Forest regression at a large scale. I want to train the Random Forest model using as many samples as possible (~100,000 samples), but it seems the Google Earth Engine can only train RF using no more than 5,000 samples. I wonder if there's a solution for this rather than split the study area.

4
  • 2
    Please be weary of the path you are on. It is a misnomer that random forests cannot overfit. If there is a model assumption in weak learners it is that there is heterogeneity in the ensemble. With the number of samples you are thinking, you may very well end up with strong correlation in the ensemble and on overfit model. Nov 29, 2020 at 23:36
  • Are your training samples distributed equally across all of your classes?
    – Aaron
    Nov 30, 2020 at 20:04
  • Hi Aaron, the samples are equally distributed.
    – Dong
    Nov 30, 2020 at 21:24
  • Thanks Jeffrey, I train random forest at a very large scale, so I think 5,000 samples are not enough. If 100,000 are too many, I would train the model use less samples (say 20,000). However, I failed to train the model with more than 5,000 samples.
    – Dong
    Nov 30, 2020 at 21:27

1 Answer 1

1

I have found that Earth Engine will take about 120 MB of data in the form of a .csv uploaded as a table asset. I recommend pulling the file into .csv and paring it down to this maximum amount. Also, with Random Forest, you can get a good idea of the feature importance of each feature (column in training data table). I use sklearn to find the the most important features. My data is normally 100+ features, from which I choose the most import 50 or so. With fewer columns, I'm able to get more samples (rows) into a table asset in EE.

For example, using the the entire training data set as csv, you might learn more about your data using the following function:

from pandas import read_csv
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

def find_rf_variable_importance(csv, target, drop=None, n_features=25):
    first = True
    master = {}
    df = read_csv(csv, engine='python')
    labels = list(df[target].values)
    df.drop(columns=drop + [target], inplace=True)
    df.dropna(axis=1, inplace=True)
    data = df.values
    names = df.columns

    for x in range(10):
        d, _, l, _ = train_test_split(data, labels, train_size=0.67)
        print('model iteration {}'.format(x + 1))
        rf = RandomForestRegressor(n_estimators=150,
                                   n_jobs=-1,
                                   bootstrap=True)

        rf.fit(d, l)
        _list = [(f, v) for f, v in zip(names, rf.feature_importances_)]
        imp = sorted(_list, key=lambda x: x[1], reverse=True)
        print([f[0] for f in imp[:10]])

        if first:
            for (k, v) in imp:
                master[k] = v
            first = False
        else:
            for (k, v) in imp:
                master[k] += v

    master = list(master.items())
    master = sorted(master, key=lambda x: x[1], reverse=True)
    print('\ntop {} features:'.format(n_features))
    carry_features = [x[0] for x in master[:n_features]]
    print(carry_features)
    return df[carry_features + [target]]

where drop is a list of unneaded features, n_features is the number of features you want to end up with, and target is your label.

In this way, I've been able to get over 60,000 samples with around 50 features into Earth Engine. Your mileage may vary.

Think critically about your feature selection. I have found in practice that I've had highly correlated features which adds data but not much information. Correlation analysis, as well as ensuring your training data covers a wide distribution of possible feature values feels like getting in the weeds, but can pay off.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.