How to programmatically check if the number of shapes = number of table records?

Question

I have a handful of approximately 1000 shapefiles that are corrupted (see attached error message). The shapefiles were generated from eCognition Developer 8. There is a script tool that seems to repair the shapefile once it is identified as corrupted.

Edit:

I want to create a quick script to loop through all of my shapefiles and check if the number of shapes matches the table records. I can count table records using the following:

# Name: fcCount.py
# Purpose: calculate the number of features in a featureclass

# Import system modules
import arcpy
from arcpy import env

env.workspace = "C:/data"
Sample = "MyShp.shp"
result_dbf = int(arcpy.GetCount_management(Sample).getOutput(0)) 
print result_dbf

I would ultimately like to create some sort of logic check such as:

if result_dbf = result_shp:
    pass
else:
    print "There is a problem with" + str(Sample)

How can I count shapes directly without accessing the .dbf file? Or, in other words, what is the best way to programmatically check if the number of shapes matches the number of table records?

Answer 1

What about using pyshp? I installed it with pip and what I tried below is pretty much straight out of the README:

>>> import shapefile
>>> sf = shapefile.Reader("/Users/chad/CoalOutcrops.shp")
>>> shapes = sf.shapes()
>>> len(shapes)
33732
>>> records = sf.records()
>>> len(records)
33732
>>>

Unfortunately (or maybe fortunately?) I don't have any jacked-up shapefiles to test to see if no. of shapes can != no. of records.

Wait just a minute, I do now have a jacked up shapefile thanks to Kirk's idea in the comments below. I backed up the dbf, made a copy of the entire shapefile, deleted some features, then renamed the backed-up dbf back to the original, and lo and behold, the number of shapes < number of records:

>>> sf = shapefile.Reader("/Users/chad/CoalOutcrops.shp")
>>> records = sf.records()
>>> len(records)
33732
>>> shapes = sf.shapes()
>>> len(shapes)
33721
>>>

Answer 2

From the sound of your question, it seems like all you really want to do is determine whether or not a shapefile has issues with it (in this case, mismatched records). If you just need to identify those with issues, you don't actually need to count the records in the DBF and Shapefile to determine if it is in error. Here's why:

If you try to run the GetCount function on a shapefile that has different record counts, it will fail with the error:

ERROR 000229: Cannot open . Failed to execute (GetCount).

Since the GetCount function fails in this scenario, and all you want to do is identify the shapefiles in error, you can catch this with a try/except clause in your code, instead of the if/else you were previously attempting to use.

I took the liberty of adding the "List FeatureClasses" code and loop so that you could test all the FC's in your workspace without manually having to test each one.

# Import system modules
import arcpy
from arcpy import env

env.workspace = "C:/data"

fcList = arcpy.ListFeatureClasses()

for fc in fcList:
    try:
        result_dbf = int(arcpy.GetCount_management(fc).getOutput(0))
        print fc + ": " + str(result_dbf) + " records"
    except:
        print "There is a problem with: " + str(fc)

Answer 3

The shapefile format is documented. I would guess the number of records in the shp file does not correspond to the number of records in the dbf file.

The shp file format is documented here. So you could write a program to count the number of shapes. The dbf format is documented in many places and you should be able to find samples for counting rows, e.g. here.

Answer 4

The attached script loops through a directory and checks if the number of shapes matches the number of records for each shapefile.

import arcpy, os, shapefile
from arcpy import env

env.workspace = r"C:\path\to\shapefiles"
Dir = env.workspace

fclist = arcpy.ListFeatureClasses()

for fc in fclist:

    myfc = os.path.join(Dir, fc)
    sf = shapefile.Reader(str(myfc))
    shapes = sf.shapes()
    shape_total = len(shapes)
    records = sf.records()
    record_total = len(records)

    if shape_total != record_total:
        print "There is a problem with " + str(fc)
    else:
        print str(fc) + " passed"

Answer 5

Using the check geometry should get you through the first step.
Onus
Repair Geometry will allow you to select the order and priority of problem you want to repair for.
here are some other older version links. When you run shapefile checker, then you finish with rebuild dbf?
That is the step that creates the records to match. One of two things has occurred to cause the error.

1. The shp has an object (spatial) that has been deleted/dropped by another software/process.
2. The dbf has a record that is referencing null geometry.
   Several things can cause this.
   The shx is actually the index between the two.
   Counting shapes without counting dbf records is only half of the solution.

Answer 6

Looking at the wikipedia article on shapefiles, the .shx file should contain an index on the .shp file, not on the .dbf file. So it might be necessary to check if .shx and .shp fit together.

It is possible to open a shapefile without .dbf (meaning you have no attribute table), but a broken index will generate an error message.