1

I am writing point shape files for a Monte-Carlo predictive model. My shapefiles are exceeding 4 GB in size. I see corrupt attribute table entries.

Is this a result of the large file size?

I'm using ArcGIS Pro to examine the attribute tables. I've been searching my C++ code for other memory overwrites (clobbering) and other possible problems.

Could it just be a matter of trying to write too big a file?

  • 1
    Have a look at the size issues in gdal.org/drv_shapefile.html. – user30184 Feb 14 at 22:18
  • I have 7G dbf file from ESRI Shapefile (shp + dbf + shx + prj is 11Gb). Got it on data.nextgis.com - the whole Germany. No problem in QGIS and ArcGIS too. It seems to me the 2/4Gb limit is already solved in current versions of software. – Dmitry Baryshnikov Feb 15 at 7:25
3

Yes, the limit you're hitting is part of the .dbf format, where the attributes are stored and for the overall shapefile there is an inherent limit size of <=4GB.

So you'll have to consider another flat file data store type like geopackage or spatialite (these are just examples). Keeping in mind other end users of the data and what their software can support opening up.

1

In the original DBF implementation, the limit was 2 GB. The 4-GB limit is an OGR "innovation". There are many DBF SDKs/libraries/tools out there that can't handle 4-GB DBFs.

The corruption you experienced is most probably caused by the software not checking the 2/4-GB limit.

As SaultDon had correctly answered, GeoPackage, or Spatialite, or PostGIS/PostgreSQL is a better choice for large data. (Remember to index them.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.