7

I have point shapefiles in 50,000 different folders (each shapefile file name is unique) - about 33,400 shapefiles (not all folders will have a point shapefile). I like to open the point shapefile in each of the folder into QGIS Desktop 2.2.0 map frame and then hopefully run a merge on them to get a single point shapefile layer. I tried drag-dropping from Windows Explorer to QGIS, and QGIS froze initially and continue to add the point shapefiles for several hours and didn't seem to stop. I found the Load Them All plugin (http://goo.gl/HbJhhS) that seem to specify the feature type and also specify a string filter, this plugin also didn't seem to work...

I was thinking that had I planned well, I could've automated and copied the point shapefiles into another single folder as soon as they were created and I could 've dragged-dropped into mapframe from this single folder. I don't know if this would've had worked. My analysis run took 18 days to run, so I don't want to re-run just to do that. I tried copy-pasting with Windows Explorer and it hung, may be because it was 100,000+ files...

Would you know any trick to do this?

4
  • 2
    I recommend you use, though cannot direct how, Python!!! Aug 13, 2014 at 21:27
  • Do the shape files, that are points, have a unique name? Like _pt.shp for example that distinguishes them from other shape files that aren't points?
    – SaultDon
    Aug 13, 2014 at 22:29
  • Shapefiles are named: Poly1_Image1.shp, Poly2_Image1.shp, Poly3_Image1.shp, Poly4_Image4.shp, Poly5_Image5.shp. Aug 13, 2014 at 23:18
  • @ArunGovind What problem did you have with Load Them All? The plugin was actually built for this kind of scenarios. Jan 28, 2015 at 21:40

3 Answers 3

9

I advise merging your shapefiles into a single folder (using a program or search/copy/paste or a small script such as the one Mark C recommends in the comments). Then use this rather elegant bit of code from gis-programming.com:

as above, use: ogr2ogr merge.shp file1.shp to create a shapefile merge containing the data of file1

then:

 for %f in (*.shp) do ( 

 ogr2ogr -update -append merge.shp %f  -f “esri
 shapefile” -nln merge 

 )
4
  • That is really very elegant! Aug 13, 2014 at 22:12
  • It would be nice to bring the shapes into a single folder, but if he uses a GUI, there could be a problem when the files exceed 65536 or something like that. There are ~50,000 shape files to deal with in this case. It would be easier to recursively search each folder and test for its geometry type instead.
    – SaultDon
    Aug 13, 2014 at 22:32
  • See HERE for a Dos Batch File to copy all the files into one common directory Aug 14, 2014 at 1:35
  • This is a good answer, @ArunGovind. And if it's germane, I additionally recommend reviewing the answers in this question if it's worthwhile to add a new field to preserve/distinguish which points originally belonged to each of your merged shapefiles.
    – elrobis
    Aug 14, 2014 at 15:36
2

Here is a simple solution for those facing a similar problem in R:

library(sf)
library(magrittr)

# find all shape files
  all_shape_addresses <- list.files(path="R:/root_folder", pattern = ".shp", full.names =T, recursive = TRUE)

# read all files into a list
  files_list <- lapply(all_shape_addresses, st_read)

# Merge all files into a single object
  single <- Reduce(st_union, x)

# save single shape file
  st_write(single , dsn = "single.shp", layer = "single.shp", driver = "ESRI Shapefile")

I'm not sure how the speed would compare to the solutions presented in the other answers. The performance of the solution in R might be restricted due to memory constraints in case there are too many and too large shape files.

1

I got this to work. But thank you to everyone who answered and their awesome ideas! I'll explore each of them.

I used Global Mapper which has the option to search directories and its sub-directories for selected file types (File > Open all files in a directory tree). It took around 12 hours to load the shapefiles from 50,000 sub-folders and then I selected point or polygon layer and separately exported them as a new vector layer. The export took 15-30 minutes for each layer.

All points and polygons

Selected points and polygons zoomed in

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.