I am currently doing an analysis on a quite huge data set - 250,000 polygons. I have figured out the general Moran's I for a linear model, but when I was trying to do a spatial lag or spatial error model, the error occurs:

Error: cannot allocate vector of size 445.6 Gb

I am using R for the analysis, and I tried to use a sparse matrix but only "LU" sparse method can succeed, and the result is not really good enough.

Any solutions for that, perhaps using spatial regression based on Hadoop? Our lab can spend some budget on that, but won't be large.

  • Can you post some code and explain how the result if sparse is not good enough? Interesting question though – John Powell Jan 3 '16 at 10:37
  • Sorry about that cause I got my result about 15 days ago. With spatial error model there are over 50 warnings but at least I could get a result, but with spatial lag model, some coefficients are NA, which is so strange. I may need to check if I still have the output file. – Elvis Wu Jan 3 '16 at 10:44
  • It is a statistical model so, why not subsample the data? With an n that size you are likely representing the population, making a subsampling approach warranted. – Jeffrey Evans Jan 3 '16 at 15:28
  • Our lab want to explore the difference between sampling and using the whole dataset. That why I am in charge of this project.. – Elvis Wu Jan 3 '16 at 22:38
  • Have you considered using packages that transfer the object creation from RAM to HDD? While this is obviously slower during the calculation It should stille be faster as a one-off solution than buying/building/configuring a hadoop cluster. – Kersten Jan 4 '16 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.