1

While trying to merge 2 large point shapefiles using the merge shapefiles tool under the data management.

These point files have over 20Gb and have the same coordinate system.

Problem is, that the resulting shapefile had a huge lapse of data, basically a hole without any kind of points populating the location.

Is QGIS unable to process properly very large files or is there a more efficient way of doing this?

The merge is needed in order to run a query on the data (points in polygon would be useful)

  • Impossible to answer without having test data of the same scale. However, I do not understand how you can have a 20 gigabyte shapefile. The dbf part can theoretically be however big but there is 2 GB or 4 GB or 8 GB limit with other parts, depending on implementation. – user30184 May 25 '16 at 12:23
  • The dbf part is the huge part since the data was bought in csv format – Luffydude May 25 '16 at 12:32
  • Then it might work if .shp parts together will not grow over 2/4/8 GB. There is another tool in Processing toolbox - QGIS Geoalgorithms - Vector general tools - Merge vector layers. Have you tried it? – user30184 May 25 '16 at 12:48
  • That was the one I used – Luffydude May 25 '16 at 12:52
  • But you wrote "using the merge shapefiles tool under the data management". Have you tried that one then? – user30184 May 25 '16 at 13:06
3

I think first of all you'll have to split the csv-files you bought in order to generate 'small' shapefiles of valid size. Then possibly it is a suitable approach for you to load your shapefiles into a PostGIS database and do the merge in the database? Thus you shouldn't have anymore problems with file size an the processing would be faster too.

1

ShapeFiles have size limits:

.shp maximum 8gb

The other parts, for example .dbf do not, but are recommended to not exceed 2gb.

http://www.gdal.org/drv_shapefile.html

  • One of the dbf files alone has 26Gb – Luffydude May 25 '16 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.