# How to programmatically check if the number of shapes = number of table records?

I have a handful of approximately 1000 shapefiles that are corrupted (see attached error message). The shapefiles were generated from eCognition Developer 8. There is a script tool that seems to repair the shapefile once it is identified as corrupted.

Edit:

I want to create a quick script to loop through all of my shapefiles and check if the number of shapes matches the table records. I can count table records using the following:

# Name: fcCount.py
# Purpose: calculate the number of features in a featureclass

# Import system modules
import arcpy
from arcpy import env

env.workspace = "C:/data"
Sample = "MyShp.shp"
result_dbf = int(arcpy.GetCount_management(Sample).getOutput(0))
print result_dbf


I would ultimately like to create some sort of logic check such as:

if result_dbf = result_shp:
pass
else:
print "There is a problem with" + str(Sample)


How can I count shapes directly without accessing the .dbf file? Or, in other words, what is the best way to programmatically check if the number of shapes matches the number of table records?

-
I imagine the file can be viewed, but are each of the items in the attribute table represented by an object? that is what the sbn file takes care of. regardless of whether it displays the number doesn't match. shapefilerepairer is what I use. –  Brad Nesom Jul 1 '13 at 15:02
Decompiling the script might be useful, but wow that is some old code! I'm honestly surprised it still works on today's shapefiles. –  Paul Jul 1 '13 at 15:09
@Brad I updated the post to make corrections. The .sbn error is a different issue I have been having and is unrelated to this problem. –  Aaron Jul 1 '13 at 15:10
@Brad When I run a corrupted file through the Shape Checker, it reports: "Not enough records in dbf file - adding blanks". –  Aaron Jul 1 '13 at 15:13

What about using pyshp? I installed it with pip and what I tried below is pretty much straight out of the README:

>>> import shapefile
>>> shapes = sf.shapes()
>>> len(shapes)
33732
>>> records = sf.records()
>>> len(records)
33732
>>>


Unfortunately (or maybe fortunately?) I don't have any jacked-up shapefiles to test to see if no. of shapes can != no. of records.

Wait just a minute, I do now have a jacked up shapefile thanks to Kirk's idea in the comments below. I backed up the dbf, made a copy of the entire shapefile, deleted some features, then renamed the backed-up dbf back to the original, and lo and behold, the number of shapes < number of records:

>>> sf = shapefile.Reader("/Users/chad/CoalOutcrops.shp")
>>> records = sf.records()
>>> len(records)
33732
>>> shapes = sf.shapes()
>>> len(shapes)
33721
>>>

-
Maybe try making a copy of the shape file (files, actually). Then in the copy delete some features. Then replace the original dbf with the copied dbf (which has had some rows deleted). –  Kirk Kuykendall Jul 3 '13 at 17:17
@KirkKuykendall - your idea worked, see edits. Thanks. –  Chad Cooper Jul 3 '13 at 17:28
No problem. If you ever need me to corrupt some more data, just let me know. –  Kirk Kuykendall Jul 3 '13 at 17:31
Thanks for the help @Chad, the shapefile module did the trick. I posted the final script used to successfully check my shapefiles. There were about 50/1000 corrupted files. –  Aaron Jul 5 '13 at 14:38

From the sound of your question, it seems like all you really want to do is determine whether or not a shapefile has issues with it (in this case, mismatched records). If you just need to identify those with issues, you don't actually need to count the records in the DBF and Shapefile to determine if it is in error. Here's why:

If you try to run the GetCount function on a shapefile that has different record counts, it will fail with the error:

ERROR 000229: Cannot open . Failed to execute (GetCount).

Since the GetCount function fails in this scenario, and all you want to do is identify the shapefiles in error, you can catch this with a try/except clause in your code, instead of the if/else you were previously attempting to use.

I took the liberty of adding the "List FeatureClasses" code and loop so that you could test all the FC's in your workspace without manually having to test each one.

# Import system modules
import arcpy
from arcpy import env

env.workspace = "C:/data"

fcList = arcpy.ListFeatureClasses()

for fc in fcList:
try:
result_dbf = int(arcpy.GetCount_management(fc).getOutput(0))
print fc + ": " + str(result_dbf) + " records"
except:
print "There is a problem with: " + str(fc)

-
Thanks Ryan, this is a good alternative to Chad's solution and also does the trick. –  Aaron Jul 5 '13 at 14:50

The shapefile format is documented. I would guess the number of records in the shp file does not correspond to the number of records in the dbf file.

The shp file format is documented here. So you could write a program to count the number of shapes. The dbf format is documented in many places and you should be able to find samples for counting rows, e.g. here.

-
Rows in a dBase file can be counted in two ways: (1) a record in the header stipulates how many rows it contains and (2) subtract the header length from the total file length (in bytes) and divide by the record length (equal to one plus the sum of lengths of the fields). It's usually a good idea to do both in case the file is physically truncated. Regardless, even when the counts match, the .shp and .dbf files are almost useless without the .shx file, which indexes into the .shp file. So a quick check of the count of .shx records might be better than reading the entire .shp file. –  whuber Jul 3 '13 at 17:35

The attached script loops through a directory and checks if the number of shapes matches the number of records for each shapefile. Problem solved...thanks for the help everyone.

import arcpy, os, shapefile
from arcpy import env

env.workspace = r"C:\path\to\shapefiles"
Dir = env.workspace

fclist = arcpy.ListFeatureClasses()

for fc in fclist:

myfc = os.path.join(Dir, fc)
shapes = sf.shapes()
shape_total = len(shapes)
records = sf.records()
record_total = len(records)

if shape_total != record_total:
print "There is a problem with " + str(fc)
else:
print str(fc) + " passed"

-

Using the check geometry should get you through the first step.
Onus
Repair Geometry will allow you to select the order and priority of problem you want to repair for.
here are some other older version links. When you run shapefile checker, then you finish with rebuild dbf?
That is the step that creates the records to match. One of two things has occurred to cause the error.

1. The shp has an object (spatial) that has been deleted/dropped by another software/process.
2. The dbf has a record that is referencing null geometry.
Several things can cause this.
The shx is actually the index between the two.
Counting shapes without counting dbf records is only half of the solution.
-
Unfortunately, repair geometry does not clear the error. –  Aaron Jul 3 '13 at 12:00

Looking at the wikipedia article on shapefiles, the .shx file should contain an index on the .shp file, not on the .dbf file. So it might be necessary to check if .shx and .shp fit together.

It is possible to open a shapefile without .dbf (meaning you have no attribute table), but a broken index will generate an error message.

-
By whom is it "not allowed"? It is possible to recover all the feature information just from the .shp file. –  whuber Jul 3 '13 at 17:31
By the software that expects a well functioning index. Not the right terms, I changed the answer a bit... –  AndreJ Jul 3 '13 at 18:11