0

I'm trying to intersect large datasets in OGR and python using SQL queries like ST_Intersects and ST_Contains. These operations run very fast in ArcMap but I'm designing a system to automate them, hence using OGR and python. My code is taking forever to run (days). I've read recently that spatial indexing will help speed up these operations. I can generate these files by indexing the shapefiles in ArcMap, which generates .sbx and .sbn files. If those files accompany my .shp .dbf .shx and .prj files in the corresponding directories, will the speed of the operations benefit?

Alternatively, will the operation speed increase if the shapefiles are loaded in to a postGIS database instead of read in via the file system?

Improve this question
1
  • 2
    Esri never published the layout of the .sbn files (.sbx is just a VFILE access complement, same as the .shx). You can try and get the answer faster than asking, but I suspect that you'll need to use a format where the spatial index is part of the access code (SQL database or file geodatabase).
    – Vince
    Commented Dec 3, 2015 at 16:06

2 Answers 2

Reset to default
3

The answer is Yes. The secret of .sbn and .sbx files has been reverse engineered and GDAL supports them since version 1.10. Read

http://erouault.blogspot.fr/2012_06_01_archive.html

GDAL manual page http://www.gdal.org/drv_shapefile.html gives also that information:

Starting with OGR 1.10, it can also use the ESRI spatial index files (.sbn / .sbx), but writing them is not supported currently.

However, your problem may mean that the OGR operations which you are using do not utilize spatial index. It does not happen automatically but someone must write the code. The -spat option in ogr2ogr for sure does support both shapefile spatial index variants.

Improve this answer
1
  • 1
    I should read all I quote :-)
    – Ian Turton
    Commented Dec 3, 2015 at 16:50
2

The sbn (and sbx) mystery was solved in 2012 : Geospatial Python: SBN Mystery - Solved!

The GIS community now has access to both the sbn and sbx file format as well as the algorithm for grouping features in a shapefile into "spatial bins".

The first result of the reverse engineering of the .sbn format is the work of Marc Pfister (Tools to create ESRI's .sbn spatial indexing format) and after, support was added in GDAL >= 1.10, (GDAL/OGR using Shapefile native .sbn spatial index) as ser30184 says.

But it is a simple spatial index tree and there are many others solutions in Python (Pypi: Spatial index, Fastest way to join many points to many polygons in python and many others examples in GIS SE)

The fastest way at the present time is the combination of Fiona (another Python wrapper of the OGR C/C++ library) , Shapely (Python wrapper of the C++ Geos library) and Rtree ( Python wrapper of the C library libspatialindex).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.