Timeline for Using 'sp' method to subset SpatialPointsDataFrame with SpatialPolygon returns error in parLapply()
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 14, 2016 at 6:26 | history | edited | fdetsch | CC BY-SA 3.0 |
Removed non-required text and code.
|
| Apr 14, 2016 at 6:24 | comment | added | fdetsch |
A classic win-win situation. I didn't know that exporting chnks is not necessary, thanks for that!
|
|
| Apr 13, 2016 at 18:37 | vote | accept | Cotton.Rockwood | ||
| Apr 13, 2016 at 18:37 | comment | added | Cotton.Rockwood |
I confirmed that I do not need to export chnks since it is passed to parLapply. Doing so only takes up memory on the workers unnecessarily. The solution to my issue was explicitly loading the package rgdal (and dependency sp) to the workers with clusterEvalQ(cl, library(rgdal)). Thanks!
|
|
| Apr 13, 2016 at 15:31 | comment | added | Cotton.Rockwood |
Also, do I need to export chnks to the workers even though it is the list being passed to parLapply? It would seem counterproductive to have to send the whole list object to all workers when they only need the element they will process.
|
|
| Apr 12, 2016 at 23:13 | comment | added | Cotton.Rockwood |
Thanks @fdetsch! That was very helpful. I am doing this work on Ubuntu and had missed that I needed to use clusterEvalQ() in conjunction with parLapply() to load packages. It still seems odd that sp looked to be loading on the workers, but not the methods functionality. Is that because the sp methods are stored in package methods as @Edzer Pebesma mentioned in the comment above, not in sp?
|
|
| Apr 12, 2016 at 7:14 | history | answered | fdetsch | CC BY-SA 3.0 |