I have the daunting task doing raster analysis on a record set that runs into quite a few millions, and worse still is that the data is in Hive.

So far I have managed to use record set of 2 million and create a raster but, in doing so, I consumed about 15Gb of my server's RAM and took around 5 minutes to complete the whole process. I am expecting the volume of data to increase a few fold.

I have gone through the big data analysis pages of ArcGIS but that did not help me.

Can somebody suggest a better option of doing big data analysis (perhaps by not fetching the data to the code, rather the reverse)?

  • What type of raster analysis are you trying to perform? Depending on how your data is stored in Hive, you may be able to write a simple query/MR program to do the analysis right in Hadoop. – Evil Genius Jun 9 '14 at 11:10
  • i am using point to raster conversion based on the bin formula(like SUM, MEAN) input from the JS client side.Can you please elaborate little how MR can be written. – Arijit Bose Jun 9 '14 at 11:17
  • Sorry, I'm abbreviating: MapReduce. They are usually written in Java, but do not have to be. Another option that may be simpler would be to tile your data (just take small chunks and analyze them separately). – Evil Genius Jun 9 '14 at 11:26
  • Thanks for the clarification. Does esri has raster api supporting jars or framework to write map reduce jobs?I have no prior experience in gis map reduce. – Arijit Bose Jun 9 '14 at 12:59
  • 1
    The GIS Tools for Hadoop have been described in a blog posting. To try and get assistance through the Q&A offered by GIS SE, I recommend that you try to focus each Question on a single question and then if more questions come up in your mind as you read Comments that seek clarifications you can research/ask them separately. There is some good advice available on How to frame a good Question. – PolyGeo Jun 17 '14 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.