Reading Geoprocessing considerations for shapefile output by Esri, it says that the 2GB limit translates roughly to 70 million Points.
When looking at the ESRI Shapefile Technical Description, the header has a field at Byte 24 representing file length in 16-bit words that is a signed integer of 4-bytes, thus the maximum positive value it can represent is 2,147,483,647. This number is the total number of 16-bit words in the file, including the 50 16-bit words in the header.
If we start with the maximum number and remove the header, we get 2,147,483,597. This means that the number of Point features in 16-bit words should equal that number (at max).
Per the spec, a record is 8-bytes and is thus 4 16-bit words. The content length field of a record represents the length of a record's content in 16-bit words. A Point feature takes a maximum of 20 bytes and is thus 10 16-bit words. Therefore, each Point feature takes a total of 14 16-bit words (4 from the record header, 10 from the Point record).
From this, how is a rough maximum of 70 million Point features derived?
It would appear the 2GB limit was Esri imposed. If we assume the following:
limit = (1024^3) * 2 = 2147483648 bytes limit - 100 = 2147483548 bytes (header removed) limit / 28 bytes = 76695841 Points
So it actually has nothing to do with Byte 24 but namely the .SHX 32-bit offsets.
However, Byte 24 in the main file and the .SHX offsets represent the number of 16-bit words in the file. Assuming our was all points, and Byte 24 was maxed at 2,147,483,647 then that implies that there are
2*((2^31)-1) bytes in the files for addressing. This would put the physical limit of the file in terms of memory addressing at 4GB. That is to say:
byte_24 = (2^31) - 1 = 2147483647 total 16-bit words byte_24 - 50 = 2147483597 total 16-bit records byte_24 * 2 = 4294967194 total 32-bit (4-byte word) file size (byte_24 * 2) / 1024^3 = 3.99GB
So there is nothing preventing a file (other than conformance) from growing beyond the 2GB limit in terms of the .SHP or .SHX files; the .DBF (dBase) may have a different impact. This could even be increased to 8GB if unsigned integers were used (replace
(2^32) - 1).