I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive.
So far I have managed to use record set of 2 million and create a raster but, in doing so, I consumed about 15Gb of my server's RAM and took around 5 minutes to complete the whole process. I am expecting the volume of data to increase a few fold.
I have gone through the big data analysis pages of ArcGIS but that did not help me.
Can somebody suggest a better option of doing big data analysis (perhaps by not fetching the data to the code, rather the reverse)?