0

My objective is to create a buffer of 1000 m for enode. I used the following query.

CREATE external table enodebuffer (shape binary);
INSERT OVERWRITE TABLE enodebuffer 
    SELECT ST_Buffer(enodeaggregate.shape,100) FROM enodeaggregate;

I am able to retrieve the columns and their values (in binary) on hive and also able to get an accurate count of the features. I am also able to get a json fil after using copu from HDFGS (esri git hubtool for hadoop)

However when I use Json to features tool of github I get the following error:

error

I get the same error whenevr I use the ST_Buffer geometry query. I also tried creating multiple columns with the shape column but to no use.


I took into account all your suggestions but I am not able to get the desired result when I want to create a buffer for 1000's of points. 1) SO for the table, as you had suggested

create external table enodebuffer1 (EnodeBStatus string,shape binary) ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde' STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';

2) Then I inserted the values in the table enodebuffer1 using the select query as below

Insert into table enodebuffer1 select enodeaggregate.EnodeBStatus,ST_Buffer(enodeaggregate.shape,1000)from enodeaggregate;

3) I got the result showing that the data has been loaded into the table succesfully, and am also able to retrieve the values using select and limit query.

4) Now the main part is I want to able to see the above results in ArcMap. So I used the esri github tools for hadoop COPY FROM HDFS and JSON TO FEATURES. Copy from HDFS was successful, no error. But json to features it shows the following error (I hope you can see the images)

JSON to features toolbox Error results on ArcMAP

5) So as Evil Genius had suggested I had tested it with 100 points and it worked fine.Possibly the error is it cannot handle too many vertices. But there has to be a solution because we are planning to migrate terabytes of data into HDFS and using multiple geometry queries to retrieve the desired results.

4
  • 2
    Please include error messages as text rather than links to pictures. – PolyGeo May 4 '16 at 11:48
  • Have you tried testing this on a small set of data? Offhand, it looks like a memory issue, which could mean something along the lines of a feature having more vertices than the conversion script can handle. – Evil Genius May 4 '16 at 12:05
  • Hi,Yes i tried it with few points, and it worked. thanks a lot. – Srinivas Santhanam May 4 '16 at 12:30
  • This issue is cross posted here, but with a different error message? – Mike Park May 4 '16 at 15:03
2

This is an issue with using the Enclosed JSON option when running JSON to Features. When this option is selected, the entire file is read into memory (json_fc = json.load(json_file)). You will want to use the Unenclosed JSON option, which means that you will also need to create your Hive table using the UnenclosedJsonInputFormat.

For Example:

CREATE EXTERNAL TABLE IF NOT EXISTS enodebuffer (shape binary)                                         
ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde'              
STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
1
  • With Spatial-Framework-For-Hadoop v1.2+, use .....EsriJsonSerDe and ....UnenclosedEsriJsonInputFormat. – Randall Whitman Jul 17 '17 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.