0

I have some shapefiles. They have associated .dbf files with them that contain many many rows of data corresponding to objects that were clipped from an earlier file.

How can I use Python to clean up the .dbf file to remove the orphaned entries?

<edit 1> I don't think my files are corrupt. They just contain features with no extents along with the extant features.

I can:

  1. open the file in QGIS and manually select extant features using the "select features by area" tool
  2. Open the attribute table
  3. Invert the selection
  4. delete the orphaned features
  5. re-save

What I can't do, is do this in any reasonably automated way, such as pyQGIS

Simply opening and saving the does not remove the orphaned features

<edit 2> I tried to run a query to generate non NULL entries in a column of the attribute table but can't seem to generate non NULLS for the extant features. - I'll keep trying. If I get that to work I can look up a method to select on that and then delete the "non" features

6
  • 2
    In Shapefile the .shp and .dbf part must have the same number of element and they are linked only by order, editing the dbf will result in corrupt data (or in your case probably won't fix already corrupt data)
    – J.R
    Apr 18, 2023 at 9:34
  • @J.R Good to know directly editing the .dbf wouldn't work to solve this problem.
    – Mr Purple
    Apr 18, 2023 at 9:37
  • 2
    You can resolve whether the data is corrupt or just has Nil geometries by using a GIS application to access the data. If it's just Nils, you can use that app to delete them.
    – Vince
    Apr 18, 2023 at 12:45
  • It's unclear if the .dbf is just associated with the shapefile of part of the shapefile. You can't edit the dBase which is part of the shapefile outside of a shapefile-aware application. But if it's just ancillary records, delete at will.
    – Vince
    Apr 18, 2023 at 12:55
  • 2
    You tried to give more information about your case by editing my answer. Write a comment instead. However, by the information that you gave it seems that you do not have "orphaned" features. You have proper features with NULL geometries,. You can certainly find features with NULL geometries with Python and then delete them, but unfortunately I do not know how.
    – user30184
    Apr 19, 2023 at 7:03

1 Answer 1

1

If you really have more rows in the .dbf than there are features in the .shp and .shx part, then your data is probably corrupted.

When a feature is deleted from a shapefile it is first just marked as deleted into the dbf file. There are still the original number of features in the .shp and .shx parts, but because the software knows that they are deleted they are not read.

The procedure that re-writes the data and removes features permanently from all parts of a shapefile is called "repack". See the GDAL driver documentation https://gdal.org/drivers/vector/shapefile.html

The OGR shapefile driver supports rewriting existing shapes in a shapefile as well as deleting shapes. Deleted shapes are marked for deletion in the .dbf file, and then ignored by OGR. To actually remove them permanently (resulting in renumbering of FIDs) invoke the SQL ‘REPACK ’ via the datasource ExecuteSQL() method.

REPACK will also result in .shp being rewritten if a feature geometry has been modified with SetFeature() and resulted in a change of the size the binary encoding of the geometry in the .shp file.

Starting with GDAL 2.2, REPACK is also done automatically at file closing, or at FlushCache()/SyncToDisk() time, since shapefiles with holes can cause interoperability issues with other software.

For Python I would recommend the ExecuteSQL() method. But if the features are already wiped away from .shp and .shx but still present in .dbf then I fear you have a problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.