Explaining difference between shx and shp files of shapefile?

Question

I am searching for a more in depth explanation of the differences in shp and shx files to little avail. I mean beyond the 'shp contain geometry - shx contains an index of the geometry'.

The reason I ask, is because while working in QGIS recently, I have made two observations that led to questions in my mind around the exact differences in these file extentions:

1. QGIS can open and display both shx and shp, and each file appears to be more or less identical in their output (display),
2. but not exactly so - I have noticed that sometimes the matching shx/shp files display slightly 'off-kilter' relative to each other. It doesn't appear to be a projection issue, they simply don't draw in the exact same location as each other.

These observations made me curious as to why these differences in display exist, and why QGIS can open and operate the shx in the same manner as the shp, when previously my understanding was that the shp is the 'master' file if you will, but requires .dbf and .shx to function correctly as a single, whole entity.

Answer 1

The definitive reference on the shapefile format is the ESRI Shapefile Technical Description.

It is misleading to describe the shx as being an "index." Instead, it is the direct access offset file. There is no data in the shx, only a clone of the first hundred bytes of the shp (with the length block in bytes 24-27 sized for the shx length) followed by record number and offset to the starting byte of that record in the shp. The only location for attributes is the dbf (which is standalone -- despite "knowledge" to the contrary, the shx does not tie the shp and dbf, only record number does that).

It is possible for shapefiles to have "gaps" in the shp which make the shx indispensible, but in practice Esri tools will rewrite the entire shp and shx so that any gap created by editing records is removed. Under most conditions, it is possible to recover the shx contents if it goes missing; the same cannot be said for the shp or dbf.

The naming of shp and shx is an artifact of the VFILE variable width direct access module of the PrImeOS operating system, first ported by Esri to Unix, VAX/VMS, Data General, and IBM, then to Microsoft Windows. The sbn/sbx spatial index pair shares the same naming convention (though these are not documented within the shapefile specification). Within the original VFILE FORTRAN library, only the base file was named, and the offset file with an x terminal character just appeared at file creation.

Answer 2

Shx certainly has nothing to show on a map alone as you can read from the specification https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf.

You are indeed right in that it is possible to open shapefile by selecting the .shx part with QGIS 3.0.3 but I can't see any difference on the location. I believe that shapefile is still opened through the same path and the result is the same. If you can make a test case for reproducing the issue of different placement of the geometries please create a QGIS ticket about the issue.

If you delete .shp part you'll see that QGIS does not open .shx alone.

Answer 3

.shx is the index for a .shp file.

You need both to be able to open the .shp file. If you open a .shx file in QGIS, it opens the .shp file. If they are off, there is a projection issue.

A ShapeFile needs three components:

1. .shp - stores the geometry
2. .shx - stores the index
3. .dbf - stores the attributes

A .prj file is usually recommended, which stores the projection information. There can be other files associated with a ShapeFile, but these are not absolutely needed.

Answer 4

The SHX file, an index. These are stored in binary format. This contains: File header, identical to that in the SHP file. Bounding box for each record The offset to the start of each record, its length, and the number of parts and points.