Error when creating spatial index in MongoDB

Question

I am wanting to use MongoDB for some basic spatial calculations (on some fairly large data), but I get an error when trying to create an index on the US Counties shapefile retrieved from the US Census' TIGER data. I had to do some manipulation in the terminal to get this data into MongoDB in the first place since Mongo can't ingest shapefiles directly:

$ ogr2ogr -f GeoJSON counties.geojson tl_2015_us_county.shp -t_srs EPSG:4326 # covert shapefile to geojson
$ jq --compact-output ".features" counties.geojson > counties_reformatted.geojson # reformat the .geojson because mongoimport throws an error if you don't
$ mongoimport -d mydb -c counties < counties_reformatted.geojson --jsonArray --batchSize 1 --drop # import to mongodb

I then create the index in the mongo shell with

> db.counties.createIndex( { "geometry" : "2dsphere" } )

but I get this error:

.
.
.
Edges 839 and 841 cross. Edge locations in degrees: [-99.8924200, 36.5932380]-[-99.8960610, 36.5932360] and [-99.8960600, 36.5932360]-[-99.8960630, 36.5932360]",
    "code" : 16755

This is along the border of Oklahoma and Kansas.

I have been able to get this process to work with shapefiles of Mexico's states, so it has worked in the past. Where is the error coming from? The ogr2ogr conversion? Or something wrong with the shapefile in the first place?

The counties, tl_2015_us_county.shp have EPSG:4269 by default, so I suspected there may be a problem in converting to EPSG:4326, but changing t_srs to EPSG:4269 in my ogr2ogr sciprt did not fix the problem. I need (I think) the counties in EPSG:4326 because I have a much, much bigger points layer already in this system.

Notes:

Maybe Stack Overflow would be a better place for this?

Inverted lat/lon is not the problem, and this workaround does not work for me.

Update:

I also tried using two different shapefiles (with different resolutions) from the US Census here, and I get a very similar error. When plotting the points that mongo says overlap, there is not a perfect coincidence with the points of the shapefile. This leads me to think that something is going wrong in my conversion process.

I'm guessing this could be solved by using really low resolution data, but I'm not really comfortable with that.

Also, spatial queries without the index are not really an option here. They take excruciatingly long without the index.

Version info:

os: Ubuntu 14.04
mongodb: 3.2.7
ogr2ogr4: GDAL 1.10.1, released 2013/08/26
jq: 1.3

Shapefile (in case anyone wants to replicate this):

wget ftp://ftp2.census.gov/geo/tiger/TIGER2015/COUNTY/tl_2015_us_county.zip --no-parent --relative --recursive --level=2 --accept=zip --mirror

Answer 1

I didn't work through the issue with your data but my perspective is that the error comes down to a limitation of shapefile format. Topological consistency is not enforced/inherent in the shapefile format (esri's explanation here). The Mongo spatial indexing system does require topological consistency.

Lacking the enforcement of consistency in the source format there are multiple ways for the process to go wrong including inconsistent precision in the coordinates or rounding errors in the conversion.

My answer is that you might need a GIS for what you are trying to achieve. A GIS (QGIS for example) has tools for resolving topology problems. Of course if you go all the way to a GIS you could do the analysis there as well.

Answer 2

I tried to approach this from as many angles as possible, and I finally found something that worked. Instead of converting the .shp to .geojson using ogr2ogr, I loaded the .shp into QGIS and then saved it as .geojson. Like stated previously, this file itself cannot be loaded into mongodb directly, so I used

$ jq --compact-output ".features" counties.geojson > counties_reformatted.geojson # reformat the .geojson because mongoimport throws an error if you don't
$ mongoimport -d mydb -c counties < counties_reformatted.geojson --jsonArray --batchSize 1 --drop # import to mongodb

I also found that this works without the --batchSize option. After this, I create the spatial index with

> db.counties.createIndex( { "geometry" : "2dsphere" } )

And it works!

So what was going on? I'm guessing something with the ogr2ogr conversion, but there are still some odd things happening. After converting the .shp to .geojson in QGIS, the data display in QGIS just fine (which is obvious). However, after compacting the .geojson with jq, the documents are successfully imported into mongodb but won't display properly in QGIS! Conversely, the ogr2ogr converted .geojson displays correctly in QGIS (with the proper CRS)! As stated in the question, the ogr2ogr created .geojson can also be successfully imported into mongodb, but creating the spatial index results in an error.

Comparing the two .geojson files directly (QGIS converted vs. ogr2ogr), every QGIS-converted document is a "Multipolygon", while the ogr2ogr documents are all "Polygons" unless the feature has multiple parts. It's unclear to me why the former works over the latter, though there very well could be more differences.