4

I have over 180,000 shapefiles that I have to merge (or append) into a final shapefile. I have written a python script to do so which uses the acrpy.append_management tool. However the runtime is too long.

I was thinking of maybe testing out the merge tool but I am skeptical that it will accept this many shapefiles and merge them together in a reasonable amount of time.

Does anyone have any experience in merging large datasets? Is there an efficient way to merge large amounts of shapefiles?

Improve this question
2
  • 5
    Without going into details, there is a 2 GB limit in the filesize of individual parts of shapefile. With 180000 shapefiles you probably exceed that. Use some database instead.
    – user30184
    Oct 16, 2014 at 13:42
  • I've never merged so many feature classes, BUT 1) check your field mapping before merging and 2) do NOT use shapefiles anymore
    – radouxju
    Oct 16, 2014 at 13:44

3 Answers 3

Reset to default
3

Consider using Dissolve. This will write the output to a new feature class but it may be more efficient. In my experience dissolve operations have taken less time than merge. Run a benchmark to determine which is faster.

There may be some issue with merge/dissolve wrt to performance and dissolving circular arcs (such as results of buffering points, polylines). Densifying input geometry prior to running merge/dissolve may increase performance.

You may also want to consider using software other that ESRI. Some users experience increase persomrmace using Postgres for example (see linked threads)

Measuring performance difference between Merge and Append in ArcGIS Desktop?

Improving upon ArcGIS Buffer/Dissolve performance?

Improve this answer
1

Thanks for all the great ideas.

Unfortunately I was stuck using arcGIS as getting admin privileges at my workplace is near impossible.

Basically what I ended up doing was appending 100 files at a time rather than 1 at a time. This reduced the runtime to just over 24 hours.

Improve this answer
1

First of all 180,000 shapefiles is big number as already suggested please do care about the 2GB limit and the field mapping.

Second, some time back I got some shapfiles and I was also not happy with the performance of arcpy merge/append. So I have tried pyshp and I was very happy with the performance.

Complete merge function you will find here. http://geospatialpython.com/2011/02/merging-lots-of-shapefiles-quickly.html and an updated version here http://geospatialpython.com/2014/06/merging-shapefiles-with-pyshp-and-dbfpy.html

I am assuming you don't have any Unicode characters.

Please do share your experience.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.