3

I have a random forest model that works pretty well, taking a bunch of vanilla remote sensing raster data as input. I think it could be improved with addition of some information that I currently have stored as categorical variables (for example: geological substrate, landform, etc.). Because my RF model is built using sklearn, I need to encode these categorical variables numerically.

I've been convinced after reading this and this that one-hot encoding is a bad idea with tree-based methods because it can reduce a single important feature into lots and lots of less important features--bloating the model and reducing performance at the same time. Plus it sounds like a nightmare to add 20+ new raster bands to my image stack. But I'm struggling with which of the alternative encoding techniques to use.

Label Encoding: I understand the critique that this introduces the "dog > cat > horse" problem, but it seems like decision trees are designed such that this isn't actually a problem.

Hashing Trick: As far as I can tell, this is functionally equivalent to assigning largish random numbers to each class. It's not really clear to me how this doesn't suffer from the same ordinality problem as label encoding, but the internet seems to think it's great.

My question is this: If I have a moderate number of categorical variables (less than 40), what is the best way to encode them specifically for use in a random forest?

---EDIT--- I made a vocabulary error in the original post. I am not trying to include a large number of categorical variables/features. Instead, I am trying to include a very small number of categorical variables/features (for example: lithology, landforms), each of which has a small-to-moderate number of classes/levels within my ROI (e.g. "sandstone, granite, limestone"). So the "less that 40" statement above applies to the number of classes/levels within each of a very small number of variables/features, rather than to variables per se. So that is what I am trying to figure out how to encode.

Improve this question
6
  • 1
    I would rethink the addition of numerous categorical variables. Remember that you are applying recursive partitioning with parent nodes. As a nominal variable splits information, it can smooth the continuous variables that go left and right. Spatial predictions including nominal data tend to over smooth the estimates and retain artifacts of the nominal patters associated with the input data. – Jeffrey Evans Mar 6 '20 at 22:08
  • Hi Jeffrey, thanks for the speedy reply. I can see how that would be an issue. So would you lose the categorical variables altogether? Seems like a shame to miss out on such a trove of potentially-relevant information. – LAT Mar 6 '20 at 23:06
  • 1
    Often categorical variables are a mixed blessing and a bit of a curse in spatial estimates. I find that thinking about the specific categories and collapsing them can help (eg., there are usually specific soil types that contribute to your process). I will then frequently apply moving window functions to get a fractional cover of said categories. It is about being strategic in using data such as this and not omitting it entirely. Statistically, in how it would effect your model, think in terms of combinatorics. There is usually quite a bit of spatial/aggregation error associate as well. – Jeffrey Evans Mar 6 '20 at 23:20
  • I realized I made a mix-up in my original post regarding the number of categorical variables I want to include. The number of variables/features is quite small (two or three), and each has fewer than 40 classes/levels. Just in case that changes anything. I'm editing the original post. – LAT Mar 7 '20 at 15:26
  • 1
    Categorical variables are limited to 32 levels in random forests. Even though you are conducting a classification using spatial data. This question seems better suited to Stack Overflow (stackoverflow.com) as it is not really spatial in nature but more about coding in Python/sklearn – Jeffrey Evans Mar 9 '20 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.