2

I have a shapefile containing 8,000,000 records, with a file size of 5 gigabytes. I would like to upload shapefile onto a server, but my file is too big and it gives an error.

How can I reduce the file size without affecting the records contained?

The server only accepts shp & mdb & gdb formats.

  • 1
    Welcome to GIS SE! You mention in a comment below that your server accepts SHP, MDB, and GDB. Are you using ArcGIS Desktop (ArcMap)? Have you tried importing your data into a File Geodatabase, and then running the Compact and Compress tools on it? What type of records are they? Point/line/polygon? What type of attributes - are there a lot of fields? – Midavalo Apr 16 '17 at 5:51
  • Please edit your question to include any response to comments. – Midavalo Apr 16 '17 at 5:51
  • Can you or can you not somehow alter the data? If not, this quesion is dead in the water. Please post the sizes of the .shp and .dbf files. – bugmenot123 Apr 16 '17 at 9:27
  • 3
    If the file size if either the .shp or .dbf exceeds 2.1Gb, you don't have a shapefile, you just have a file with a .shp suffix. – Vince Apr 16 '17 at 12:53
  • What software produced the shape-file? As Vince said that is not standard conform. And it would be the safest way to export it in several parts then from the original source. Everything else might corrupt the data in a way you might not even recognize at first (common problem is missmatching of dbf records to geometry objects). – Matte Apr 16 '17 at 14:29
8

The shapefile format has built-in limits on total size, where both the .shp and .dbf are limited to 2,147,483,647 bytes (2^31-1). While you can append to a shapefile so that it exceeds this file size, once you do, it's no longer a shapefile, and many shapefile reader utilities will not function with it.

This means that you probably won't be able to convert the existing dataset to a file format that supports larger data. First you'll need to choose a format (and since .mdb is also limited to 2Gb, the only choice available from your list is file geodatabase). Then you'll need to populate a table ("feature class") in a file geodatabase from the original data source (not the corrupted shapefile). From there, is advisable to "Compact" the FGDB (which will eliminate gaps between records as a result of processing), then to "Compress" it (which will make it read-only and reduce the size a bit more). Finally, you'll need to zip the directory tree with the .gdb suffix.

The file geodatabase format uses a compression algorithm on the vertices, so the storage used by an FGDB will be less than an equivalent shapefile (it also uses variable width attribute records, so if you have any string columns, the attribute component will be smaller as well). If you have unnecessary vertices in your features, you can thin line and polygon features by using the Ramer-Douglas-Peucker line generalization algorithm, as found in the ArcGIS tools Simplify Line and Simplify Polygon (Note: This requires a Standard or Advanced license).

I should point out that you may find difficulty finding a site that will permit a multi-gigabyte upload. It's also questionable that anyone would want to commit to the time it would take to download such a massive file. So you may want to review your purpose in such an upload before you invest in the time to construct such a file.

3

I may be barking up the wrong tree here but, it may worth the effort to restructure the character or text based fields decreasing their default length.

You may have text or character fields that are set to accept 254 characters when they are only populated with 32 characters, this will create a much larger database file than necessary.

In my experience, I have never seen Name fields, Address fields, SSID fields, Phone Number fields that actually require more than 100 characters as a default. That is where I would start. If you can reduce the default length of 10 fields in your database by more than half for 8,000,000 records, you may end up with a file size that is manageable.

I have done this with many .dbf files in the past by getting the maximum length of the data in the text, or character based fields whose default values were set to 254 (255) characters, but were populated with many fewer characters than that.

I would typically take the maximum length found in those fields, and add 10 characters just to be safe, and restructure those fields accordingly.

You may get a warning that you could lose data, but have never lost data doing this, and I have ended up with files that were usable in multiple applications.

1

Following the advice of jbgramm, I wrote this script to reduce text fields to their minimum to make a smaller copy of a feature class. If you have a lot of large text fields, the file will get a lot smaller. As others have recommended you may also simplify the geometry.

-5

You can use compressing archive software, like 7-zip, winrar & ... . It compress your files and it can split your files to multiple parts.

  • Thanks but server only Accept formats shp & mdb & gdb – Ali Apr 16 '17 at 5:22
  • 2
    This won't work because a 5Gb shapefile is not valid. If you don't address the real problem you're only distracting the OP. – Vince Apr 16 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.