I have a shapefile containing 8,000,000 records, with a file size of 5 gigabytes. I would like to upload shapefile onto a server, but my file is too big and it gives an error.
How can I reduce the file size without affecting the records contained?
The server only accepts shp & mdb & gdb formats.
The shapefile format has built-in limits on total size, where both the .shp and .dbf are limited to 2,147,483,647 bytes (2^31-1). While you can append to a shapefile so that it exceeds this file size, once you do, it's no longer a shapefile, and many shapefile reader utilities will not function with it.
This means that you probably won't be able to convert the existing dataset to a file format that supports larger data. First you'll need to choose a format (and since .mdb is also limited to 2Gb, the only choice available from your list is file geodatabase). Then you'll need to populate a table ("feature class") in a file geodatabase from the original data source (not the corrupted shapefile). From there, is advisable to "Compact" the FGDB (which will eliminate gaps between records as a result of processing), then to "Compress" it (which will make it read-only and reduce the size a bit more). Finally, you'll need to zip the directory tree with the .gdb suffix.
The file geodatabase format uses a compression algorithm on the vertices, so the storage used by an FGDB will be less than an equivalent shapefile (it also uses variable width attribute records, so if you have any string columns, the attribute component will be smaller as well). If you have unnecessary vertices in your features, you can thin line and polygon features by using the Ramer-Douglas-Peucker line generalization algorithm, as found in the ArcGIS tools Simplify Line and Simplify Polygon (Note: This requires a Standard or Advanced license).
I should point out that you may find difficulty finding a site that will permit a multi-gigabyte upload. It's also questionable that anyone would want to commit to the time it would take to download such a massive file. So you may want to review your purpose in such an upload before you invest in the time to construct such a file.
I may be barking up the wrong tree here but, it may worth the effort to restructure the character or text based fields decreasing their default length.
You may have text or character fields that are set to accept 254 characters when they are only populated with 32 characters, this will create a much larger database file than necessary.
In my experience, I have never seen Name fields, Address fields, SSID fields, Phone Number fields that actually require more than 100 characters as a default. That is where I would start. If you can reduce the default length of 10 fields in your database by more than half for 8,000,000 records, you may end up with a file size that is manageable.
I have done this with many .dbf files in the past by getting the maximum length of the data in the text, or character based fields whose default values were set to 254 (255) characters, but were populated with many fewer characters than that.
I would typically take the maximum length found in those fields, and add 10 characters just to be safe, and restructure those fields accordingly.
You may get a warning that you could lose data, but have never lost data doing this, and I have ended up with files that were usable in multiple applications.
Following the advice of jbgramm, I wrote this script to reduce text fields to their minimum to make a smaller copy of a feature class. If you have a lot of large text fields, the file will get a lot smaller. As others have recommended you may also simplify the geometry.
.shpor.dbfexceeds 2.1Gb, you don't have a shapefile, you just have a file with a.shpsuffix.