Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have point shapefiles in 50,000 different folders (each shapefile file name is unique) - about 33,400 shapefiles (not all folders will have a point shapefile). I like to open the point shapefile in each of the folder into QGIS Desktop 2.2.0 map frame and then hopefully run a merge on them to get a single point shapefile layer. I tried drag-dropping from Windows Explorer to QGIS, and QGIS froze initially and continue to add the point shapefiles for several hours and didn't seem to stop. I found the Load Them All plugin (http://goo.gl/HbJhhS) that seem to specify the feature type and also specify a string filter, this plugin also didn't seem to work...

I was thinking that had I planned well, I could've automated and copied the point shapefiles into another single folder as soon as they were created and I could 've dragged-dropped into mapframe from this single folder. I don't know if this would've had worked. My analysis run took 18 days to run, so I don't want to re-run just to do that. I tried copy-pasting with Windows Explorer and it hung, may be because it was 100,000+ files...

Would you know any trick to do this?

share|improve this question
1  
I recommend you use, though cannot direct how, Python!!! –  mapBaker Aug 13 at 21:27
    
Do the shape files, that are points, have a unique name? Like _pt.shp for example that distinguishes them from other shape files that aren't points? –  SaultDon Aug 13 at 22:29
    
Shapefiles are named: Poly1_Image1.shp, Poly2_Image1.shp, Poly3_Image1.shp, Poly4_Image4.shp, Poly5_Image5.shp. –  Arun Govind Aug 13 at 23:18

2 Answers 2

I advise merging your shapefiles into a single folder (using a program or search/copy/paste or a small script such as the one Mark C recommends in the comments). Then use this rather elegant bit of code from gis-programming.com:

as above, use: ogr2ogr merge.shp file1.shp to create a shapefile merge containing the data of file1

then:

 for %f in (*.shp) do ( 

 ogr2ogr -update -append merge.shp %f  -f “esri
 shapefile” -nln merge 

 )
share|improve this answer
    
That is really very elegant! –  WhiteboxDev Aug 13 at 22:12
    
It would be nice to bring the shapes into a single folder, but if he uses a GUI, there could be a problem when the files exceed 65536 or something like that. There are ~50,000 shape files to deal with in this case. It would be easier to recursively search each folder and test for its geometry type instead. –  SaultDon Aug 13 at 22:32
    
See HERE for a Dos Batch File to copy all the files into one common directory –  Mark C Aug 14 at 1:35
    
This is a good answer, @ArunGovind. And if it's germane, I additionally recommend reviewing the answers in this question if it's worthwhile to add a new field to preserve/distinguish which points originally belonged to each of your merged shapefiles. –  elrobis Aug 14 at 15:36

I got this to work. But thank you to everyone who answered and their awesome ideas! I'll explore each of them.

I used Global Mapper which has the option to search directories and its sub-directories for selected file types (File > Open all files in a directory tree). It took around 12 hours to load the shapefiles from 50,000 sub-folders and then I selected point or polygon layer and separately exported them as a new vector layer. The export took 15-30 minutes for each layer.

All points and polygons

Selected points and polygons zoomed in

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.