2

I am new to remote sensing. I am working on a project where I need to perform land use land cover classification. For such a task I want to use machine learning (python). My features contain 'BANDS' (6 bands), 'NDVI', 'NDWI', and 'NORM' extracted from the sentinel level 1C images. I did cloud masking and then interpolation (to fill the gaps) and finally, I sampled the data and split them into training and testing datasets. I have also used a mask to eliminate samples where I have '0' which essentially means 'no-data':

# remove points with no reference from training (so we dont train to recognize "no data")
mask_train     = labels_train == 0
features_train = features_train[~mask_train]
labels_train   = labels_train[~mask_train]

# remove points with no reference from test (so we dont validate on "no data", which doesn't make sense)
mask_test     = labels_test == 0
features_test = features_test[~mask_test]
labels_test   = labels_test[~mask_test]

However, to be extra sure, I checked for the NAN values using: np.isnan(train_test).any() (for both training and testing datasets) I found that my features had NAN values. Any idea how to solve this? and why do I have NAN values in the first place?

Edit: This is what my dataset looks like ( a lot of NAN values): enter image description here

1 Answer 1

1

Nodata and Nan are not the same thing. A Nodata value is a value that is assigned to certain pixels in the case there was no observation possible (for example because of cloud cover). This may be done on purpose by you or by the creator of the data that you work with.

A Nan can arise for many reasons, but most often it happens because an algorithm tries to do something on a dataset which is not possible. For example:

import numpy as np
a = np.arange(9)
b = a / 0
print (b) # will be nans or infinate (inf)

Coming back to your use case, when you calculate NDVI you might encounter that nir+red is 0. This will result in a Nan because you will divide by 0.

You can detect and deal with Nans (and Infinite) values as follows:

import numpy as np
band1 = np.random.random(10)
nodatavalue = -999
# assign values in the array to nodata, nan and inf
band1[1] = nodata
band1[2] = np.nan
band1[3] = np.inf
mask = np.where(np.logical_or(band1 == nodata, np.logical_not(np.isfinite(band1))), 1, 0)
band1_out = np.where(mask == 1, nodata, band1)
print(band1_out)
3
  • I did apply a cloud mask, but I also used linear interpolation to fill the gaps. I did also use np.seterr(divide='ignore', invalid='ignore') to suppress the "divide by zero" or "divide by NaN" warning because I noticed that I was getting that warning. The code you wrote does it only detect Nans (and Infinite values)? should I input them? using mean values or median?
    – Rim Sleimi
    May 26, 2020 at 16:06
  • The point of my second example is to detect Nans and Infinites and replace them with the nodata value of your tiff. How to deal with these nodata values is really up to you. You can fill them with a static value, with a different observation, or simply exclude them from analysis.
    – Johan
    May 27, 2020 at 7:04
  • Is it expected to lose data (rows and columns) when you use simpleImputer ?
    – Rim Sleimi
    May 27, 2020 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.