8

I am searching for a more in depth explanation of the differences in shp and shx files to little avail. I mean beyond the 'shp contain geometry - shx contains an index of the geometry'.

The reason I ask, is because while working in QGIS recently, I have made two observations that led to questions in my mind around the exact differences in these file extentions:

  1. QGIS can open and display both shx and shp, and each file appears to be more or less identical in their output (display),
  2. but not exactly so - I have noticed that sometimes the matching shx/shp files display slightly 'off-kilter' relative to each other. It doesn't appear to be a projection issue, they simply don't draw in the exact same location as each other.

These observations made me curious as to why these differences in display exist, and why QGIS can open and operate the shx in the same manner as the shp, when previously my understanding was that the shp is the 'master' file if you will, but requires .dbf and .shx to function correctly as a single, whole entity.

18

The definitive reference on the shapefile format is the ESRI Shapefile Technical Description.

It is misleading to describe the shx as being an "index." Instead, it is the direct access offset file. There is no data in the shx, only a clone of the first hundred bytes and record number and offset to the starting byte of that record in the shp. The only location for attributes is the dbf (which is standalone -- despite "knowledge" to the contrary, the shx does not tie the shp and dbf, only record number does that).

It is possible for shapefiles to have "gaps" in the shp which make the shx indispensible, but in practice Esri tools will rewrite the entire shp and shx so that any gap created by editing records is removed. Under most conditions, it is possible to recover the shx contents if it goes missing; the same cannot be said for the shp or dbf.

The naming of shp and shx is an artifact of the VFILE variable width direct access module of the PrImeOS operating system, first ported by Esri to Unix, VAX/VMS, Data General, and IBM, then to Microsoft Windows. The sbn/sbx spatial index pair shares the same naming convention (though these are not documented within the shapefile specification). Within the original VFILE FORTRAN library, only the base file was named, and the offset file with an x terminal character just appeared at file creation.

  • If you search for .shx in that document the first entry is "The suffix for the index file is .shx". It's not a link between the dbf and shp, but as I understand it more like a spatial index. – HeikkiVesanto Sep 4 '18 at 11:02
  • 1
    And later: The index file (.shx) contains a 100-byte header followed by 8-byte, fixed-length records. Figure 4 illustrates the index file organization. Logically it is analogous to database indexes a data structure that improves the speed of data retrieval operations. – user30184 Sep 4 '18 at 11:06
  • No, the shx is absolutely not a spatial index. – Vince Sep 4 '18 at 11:41
  • 1
    "..., only a clone of the first hundred bytes and record number and ...". "... the shx does not tie the shp and dbf, only record number does that". First you say 'record number' is within .shx. Later you say .shx does not tie .shp and .dbf, but that 'record number' does. If 'record number' ties the two cited formats and it is part of .shx, then why can't it be said .shx ties .shp and .dbf? It is confusing to me this part of your answer. Can you clarify? Tks. – Andre Silva Sep 4 '18 at 23:34
  • 1
    The record number is present in both shp and shx. It is not present in the dbf, only the implicit byte offset from the end of the header records (40 bytes for table, which includes a field count, and 40 additional bytes for each field) times the record size. I've written shapefile readers and writers in both 'C' and Java, and it is the loop counter variable of the program which links the shp/shx to the dbf. The shx provides linkage of record number, by offset with a recno integrity check, to the start of the shp record, nothing more. – Vince Sep 5 '18 at 1:59
12

Shx certainly has nothing to show on a map alone as you can read from the specification https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf.

You are indeed right in that it is possible to open shapefile by selecting the .shx part with QGIS 3.0.3 but I can't see any difference on the location. I believe that shapefile is still opened through the same path and the result is the same. If you can make a test case for reproducing the issue of different placement of the geometries please create a QGIS ticket about the issue.

If you delete .shp part you'll see that QGIS does not open .shx alone.

  • My mistake, I checked the same files again and it was indeed a mismatched projection. When I realigned these, the shx and shp versions of the same layer lined up and were identical as expected. Interestingly, QGIS allows both the .shx and .shp to be uploaded to the same project, and they display their own respective source path, but as pointed out by others, the output is the exact same geometries as these are indeed the same file. – Maggie Sep 5 '18 at 7:29
2

.shx is the index for a .shp file.

You need both to be able to open the .shp file. If you open a .shx file in QGIS, it opens the .shp file. If they are off, there is a projection issue.

A ShapeFile needs three components:

  1. .shp - stores the geometry
  2. .shx - stores the index
  3. .dbf - stores the attributes

A .prj file is usually recommended, which stores the projection information. There can be other files associated with a ShapeFile, but these are not absolutely needed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.