0

I'm something of a beginner to remote sensing.

I've got an NDVI time series, from 2002-2016. I've smoothed the data using an excel spreadsheet with the formula yt = (-2xt-3 + 3xt-2 + 6xt-1 + 7xt + 6xt+1 + 3xt+2 - 2xt+3)/21. Now, I don't understand that - equations baffle me - I trust it works though, because I'm told it does. Using MODIS NDVI 16 day combined data extracted from Google Earth Engine!

Unfortunately, I've got gaps in my NDVI time series, and those gaps correspond with very low or negative NDVI values in the smoothed time series.

I've tried to read about methods to fill gaps in NDVI time series. Chen et al. (2004) proposed a method (https://www.sciencedirect.com/science/article/pii/S003442570400080X). Kandasamy et al. (2013) called that an Adaptive Savitsky Golay filter. But I must be honest, the mathematics and equations serve only to confuse me! I came to Remote Sensing from a background in environmental management rather than science, so to me those papers are sadly about as useful as hieroglyphics.

Given my mathematic incompetence, could somebody please give me either:

a) an equation to paste into Excel which could accomplish Chen et al.'s (2004) method

or

b) tell me what method I should use to fill the gaps in my NDVI time series?

Essentially - how should somebody who can't turn an equation on a page into a formula in excel fill the gaps in an NDVI time series?

closed as too broad by PolyGeo May 21 at 23:01

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Which sensor are you using? Which software? – aldo_tapia May 11 '18 at 19:41
  • 1
    This really is not an analysis that should be performed in Excel and given the software's history with statistics I would honestly likely call into question a paper that used it for their analysis. Please note that the Savitzky–Golay filter is for smoothing a time-series and if the intent is filling missing values you would have to modify its recommend implementation from Chen et al., (2004). Personally, I use a local polynomial regression for filling nodata. This can easily be done in R, or perhaps Python, but would be quite difficult in Excel. – Jeffrey Evans May 11 '18 at 19:52
  • I agree with @JeffreyEvans, you posted an excellent answer for this purpose gis.stackexchange.com/a/279383/80215 – aldo_tapia May 11 '18 at 20:02
  • I understand the Savitsky Golan is used to smooth data - I simply picked out the Chen paper because it seemed like it'd be compatible with the smoothing method I'm using. Stupid reason? Sure, I guess! – Mahavelona May 11 '18 at 20:51
  • Is Excel somehow unreliable to do the smoothing? Or just the gap filling? The excel chart was given to me by the tutor which is why I've got it. I've not got R or Python on this machine and installing would be a last resort to be honest! Reason being - I've never used either software, and I'm already finding things tricky. I'll look into local polynomial regression nonetheless. Please let me know if there are other avenues I should try! – Mahavelona May 11 '18 at 20:51
2

I will just say that Excel is not a statistical software and many statistical/mathematical operators are just not available or extremely difficult to implement. This type of data is inherently an array and best analyzed in a matrix type framework. Collapsing it to a row/column flat file with coordinate pairs and then a series of values in subsequent columns, limits the type of analysis that can be performed and, Excel is not exactly tractable with large data. That said, it honestly is none of my business that you want to use a spreadsheet.

If you want to implement a Savitsky-Golay we can work through it procedurally and you can figure out how to adapt it to a spreadsheet program. I will provide my worked example in R as I have the matrix operators available to solve the problem.

First, we will simulate a sine function, with some noise added, that generally represents a periodicity pattern.

set.seed(42)
x <- 1:1000 
r  <- sin(2 * pi * (x) / 200)
noise <- r + rnorm(length(x)) / 10
head( dfs <- data.frame(r = r, noise = noise, x = x) )

Define general model parameters

f = 4  #  quartic filter (2 for quadratic)  
l = 51 # Filter length 
d = 1  # First derivative

Calculate filter coefficients for left and right window index

fc <- (l-1)/2  

Calculate polynomial terms and coefficients

X  <- outer(-fc:fc, 0:f, FUN="^") 

Derive pseudo-inverse of a matrix using singular value decomposition

s <- svd(X)
Y  <- s$v %*% diag(1/s$d) %*% t(s$u)

Filter the data through convolution function

T2 <- convolve(dfs$noise, rev(Y[d,]), type="o")  
dfs$sg <- T2[(fc+1):(length(T2)-fc)] 

Finally, we can plot results of the smoothed data (red line) on top of the noisy data (points).

library(ggplot2)
ggplot(dfs) +
  geom_point(aes(x, noise), size = 0.75) +
  geom_line(aes(x, sg), col = "red2", size = 1) +
ylab("y") + theme_bw()

Smoothed series (red line) plotted on sine function (points)

Now that we have a smooth line fit to the observed data, to fill missing data values we would just identify the location(s) of the missing values in the vector and fill them with the corresponding values (locations) from the smoothed data. For a data smoothing approach, to mitigate error and stochasticity thus, improving the signal to noise ratio, you would simply replace the entire vector with the smoothed series.

Not the answer you're looking for? Browse other questions tagged or ask your own question.