1

I am using the ESRI framework that extends the Hive syntax to support spatial types and operations.

In particular, I am trying to use an aggregated geometry function, announced in this blog post in 2013.

What I would like to do, is to generate a bunch of buffers around some points, and then dissolve them using ST_Aggr_Union.

create table buffers as select ST_Aggr_Union(ST_Buffer(tweets.geom,0.5)) as geom from tweets join grid_cnt where ST_Contains(grid_cnt.geom,tweets.geom)=true and ptcnt>6;

The query fails with a Map Reduce error:

Hadoop job information for Stage-2: number of mappers: 0; number of reducers: 0
2015-01-21 12:55:21,758 Stage-2 map = 0%,  reduce = 0%
Ended Job = job_local1494455277_0007 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:8080/
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-2:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

If I do the query without the ST_Aggr_Union function, it runs without problems.

Just to be sure, I also tried to reproduce the example in the blog post:

SELECT ST_AsText(ST_Aggr_Union(ST_Point(earthquakes.longitude, earthquakes.latitude)))
FROM earthquakes
JOIN counties_json counties
WHERE counties.name = ‘San Bernardino’
AND earthquakes.magnitude > 6
AND ST_Contains(counties.shape, ST_Point(earthquakes.longitude, earthquakes.latitude));

It fails, with the exact same error. Could it be a bug?

Improve this question
  • Interesting, so ST_Aggr_Union is an ESRI invention. – John Powell Jan 21 '15 at 12:29
  • It wouldn't surprise me if they did. Esri's ST_GEOMETRY implementation dates back to the late 90s on DB2 and Informix (SDE 3.1.1, I believe), which was ported to Oracle at 9.2 and PostgreSQL at 9.3. The standards were under development at the time, but Esri usually had one of the first compliant implementations – Vince Jan 21 '15 at 13:37
  • Which version of Hive are you running for this? – Mike Park Jan 21 '15 at 15:27
  • The latest stable release: 0.14.0 – doublebyte Jan 21 '15 at 15:46
  • Thanks. I'll try to recreate. In the meantime, can you look in the logs for more information on why the MR job failed? – Mike Park Jan 21 '15 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.