3

I didn't know which method to use cross-validation or bootstrap in maxent so I ran both and my AUC values for each method are quite different for bootstrap I have an AUC of 0.758 and for cross validation I have an AUC of 0.737. I have quite a large dataset 3000 + occurance points, and have tested different combinations of feature classes and regularisation multipliers to find that the best for my data is LQHPT and RM of 1.5.

Can anyone explain way using each of these methods results in such a difference in AUC values and which one would be the best to use?

Many thanks in advance.

3

Cross validation splits your training data into a number of groups. Maxent then calibrates a model on a number of those groups and tests it on the the groups left out (say 2/3rd's of the folds as calibration and 1/3rd as validation).

Bootstrapping is where a new sample is selected from your calibration data by randomly selecting observations WITH replacement. This means each bootstrap sample may have repeated values and is tested on the observations not selected.

They are just two different methods for estimating the uncertainty in your evaluation of predictive power of your model. Given you have a lot of occurrence observations cross validation is probably a reliable method as bootstrapping is often preferred for small sample sizes (where folds in CV are very small).

Also, the differences between their estimates are actually very similar and are telling you they have essentially the same discriminatory power.

1
  • 2
    I would point out that you can use a bootstrap in a cross validation and they are not, in fact, two different methods. The concept of CV is to derive a performance statistic by comparing a model to data that was not included in fitting said model. This can be done iteratively, via bootstrap, jackknife, etc..., to evaluate an error distribution by using different realizations of the data in refitting the model. Splitting in to groups for a cross validation is an n-fold CV. Jul 6 '20 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.