I need to load .shp files in Hive more precisely in HortonWorks sandbox through Beeswax, can anyone provide example of Hive commands or steps for Beeswax ?

  • Do you plan on dong spatial analysis once it's there? How does the geometry need to be stored (i.e. GeoJSON)? ESRI has release a very basic framework for working with spatial data within Hadoop via Hive, GIS Tools for Hadoop by ESRI. It doesn't answer your question, but may help point you in the right direction. – Evil Genius Jul 1 '14 at 19:38
  • I am planning to use simple queries against tables using UDF functions ? I have seen this framework and there is a tool called SerDe but if undurstood correct input file must be in JSON format and don't know how to convert .shp files to JSON? – Igor Jul 1 '14 at 20:00
  • What tools, programs, libraries do you have access to? (ArcGIS, QGIS, GDAL/OGR, etc..). – Evil Genius Jul 1 '14 at 21:21
  • I have qgis and postgre with postgis addon. I am skillful with java, so java hacks are also ok... – Igor Jul 1 '14 at 21:59

You should be able to export the shape file as a GeoJSON file from QGIS.

  • With the shapefile added to your map, right click on it in the legend/TOC and click on 'Save As...' in the pop-up menu.
  • Select GeoJSON as the output format

Keep in mind that while it is valid JSON, the GeoJSON output this way may not be in the exact format you are expecting. OGR (which is what QGIS is using to export the data) will put all features into a single feature collection, which may show up as a single record depending on how you are parsing it.


Check out http://thunderheadxpler.blogspot.com/2014/01/hadoop-and-shapefiles.html - in theory my ShapefileInputFormat could be used when defining a Hive table - tho never try it !

  • You might want to give links to the two libraries described in the blog post rather than just to the blog post, and describe a little bit about where each is from and what it does (ie, summarize the blog post). That or this answer might be better as a comment. – Chris W Jul 3 '14 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.