3

I have an ESRI shapefile with 1.7 million records. When loaded in QGIS the attribute table is automatically filtered; I only see 725256 records and the filter says "filtered: 725256"? Why? I tried restarting QGIS, reloading the dataset (and another similar dataset in which the same thing happens but with a different number of records filtered) I haven't successfully been able to remove this filter yet. I really need access to all features.

I'm running QGIS with GRASS 2.18.13 on Ubuntu 16.04 LTS.

Please help!

Improve this question
5
  • 4
    Welcome to GIS.SE! It does filter the same number of features on every reload of the same dataset? The files in the shapefile do not exceed 2GB? How did you create the shapefiles? I presume that the dbf exceeds 2GB or equals 2GB and contains only your 725256 records, and for the rest of the geometries in th shp file there is no related dbf entry, maybe related: groups.google.com/forum/#!topic/fmetalk/P_ilL5bnl6E And do you have the opportunity to use PostGIS to handle such amount of records?
    – Jochen Schwarze
    Oct 20, 2017 at 9:38
  • QGIS in this case does not care about either the size of the .shp or .dbf. It will handle larger ones, and also does write larger ones without telling the user, like many other programs do. Got those files from CAD-Exports a few times with larger geometries and tested larger .dbf just 5mins ago. It will however lead to geometrie reading errors in for example ArcMap.
    – Matte
    Oct 20, 2017 at 11:27
  • Yes it filters the same number of features on every reload. Thje DBF is 2.1 GB. I merged several shapefiles into one. I guess you are correct in that this is a dbf limitation. I guess I would have the possibility to use PostGIS but that would require an additional amount of time to set up and learn and for this project I don't have that. My main problem is that there are so many topological errors that I can't correct them manually and it seems like spatialite will not accept these. I'm v.cleaning like h... but there's no really good recipe for this out there. I keep having errors! Thx a lot!
    – Greforb
    Oct 20, 2017 at 11:45
  • I Selected a subset of 1.6 mill features (as I couldn't select all because of the above issue) using the hand-selection tool) and deselected all slivers from earlier operations, leaving approx 1 mill selected. I exported these. I did the same with the small remainder and finally remerged the exported polygons without slivers. I now see that for many of these exported features the attributes are missing as predicted by Jochen Schwarze. But why could I select 1.6 million, but not all 1.7 mill?
    – Greforb
    Oct 20, 2017 at 11:54
  • Regarding "I merged several shapefiles into one" : It might be worth trying to fix the topological errors and export to spatialite before merging.
    – AndreJ
    Oct 20, 2017 at 14:22

1 Answer 1

Reset to default
4

It is possible to have more geometries than entries in the attributtable in QGIS because of issues with GDAL not reading the multipolygon geometries. The shapefile is actually a geometry collection as it does not differentiate between Polygon and MultiPolygon. In some shapefiles GDAL ignores the multipolygons (maybe when the first read objects are polygon, but that is an assumption from my side). You can import them into Spatillite or Postgis as mentioned by Jochen Schwarze. Both will read all records.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.