4

I need to load some geographic data into Hadoop (Hive), and then, to be able to call some spatial functions on them using Hive.

My input data format is ESRI shapefiles (but could be another format, I could transform if necessary).

I know that ESRI has released a very basic framework for working with spatial data within Hadoop via Hive, GIS Tools for Hadoop by ESRI, but it seems that I need to have my data in ESRI JSON and I don't have ESRI tools to convert my data into that format.

Is there any alternative way to load geodata into Hadoop or is there a way to convert my ESRI shapefiles into ESRI JSON?

  • github.com/Esri/geojson-utils converts from normal to ESRI JSON. I haven't used it, but it looks good. – John Powell Dec 11 '14 at 14:08
  • Do you have a copy of ArcGIS? There is a GP tool that can convert any feature class to the correct JSON format. – Mike Park Dec 30 '14 at 23:09
  • @climbage. OP did state in question that he does not have ESRI tools. – John Powell Jan 16 '15 at 6:33
  • @JohnBarça Oops, I missed that part. There is no hard dependency on Esri JSON, it is merely one mode of transport between Hadoop and a GIS system. – Mike Park Jan 16 '15 at 18:15
  • @climbage. Good to know. I am hoping to play with spatial hadoop, such as it exists, this year. – John Powell Jan 16 '15 at 18:16
3

ST_Geometry for Hive in the GIS Tools for Hadoop, can load data from several different formats: Well-Known Text (WKT), Well-Known Binary (WKB), GeoJSON, Esri JSON, and coordinates as a list of numbers. Please see the documentation for the constructor functions.

As for shapefile, the java library to read shapefiles by Mansour Raad may be of use.

2

The GDAL ogr2ogr (http://www.gdal.org/) functions will let you convert shapefiles to any number of formats, including JSON and WKT. I've only used it to go directly to SQL Server Spatial, so I don't know the exact commands. You can go straight to GDAL: or you can get try QGIS (http://www.qgis.org/en/site/) to get an ArcMap like UI that you might be able to work from directly, but I've found that I get the best conversion results using OSGeo4W's Shell (https://trac.osgeo.org/osgeo4w/) to write commands.

It may matter what kind of shapes you're using. I've successfully imported polygons directly into SQL with ogr2ogr, but using QGIS to convert to WKT or JSON can often truncate complex polygons.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.