Why is feature extraction important?

Why do we have to reduce the feature space?

Is it only a computational problem when we use hyperspectral imagery or does this procedure improve the generalization ability of a classification system?


Feature extraction is not always a necessity: it depends on the algorithm used for the classification. Having too many features will lead to the so-called "curse of dimensonnality". A maximum likelihood classifier will be very sensitive to this, while a SVM classifier should in theory handle a large number of features without too much problem. Other classifiers are intermediate.

A typical issue is the sparsity of the training sample : if you need to estimate a distribution, you need enough training samples in all dimension, so that the total number of samples will grow exponentially with the number of dimension. Another issue is that you increase the risk of correlated features, which can cause problem if you want to invert a matrix.

In any case, adding more feature do not necessarily add more information, and information is what you need. For example, if you want to distinguish a car from a motor bike, the intensity of the color will not help you (but if your sample is too small, you could randomly select 2 red cars and two yellow motorbikes, hence conclude that everything red is a car). Feature selection is supposed to help you select the right features (in this case, the number of wheels), but it is also an extra step where you can commit errors.


Take an agricultural field as a simple example. Unless we're interested in precision agriculture, in which case characterizing the variability within the field is important, then we're more likely just interested in knowing that the patch of land designated by that field is being used to grow soy, or corn, or whatever crop. That is, from an information sense, we're more interested in the field as a single unit. That allows us to answer questions like how many fields in Ontario are growing corn or what the acreage of soy is. And that is important for estimating crop production & insurance, etc.

  • @Gabor Farkas Done. – WhiteboxDev Nov 13 '14 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.