4

Want to merge all shp files in a folder into one with a new field populated with the source filename. Not getting any errors but not getting any outputs.

Code originally taken from (How to add field with filename when merging shapefiles with ogr2ogr?)

# merge_shps.py
import os    
path = "C:\TEMP\SWS_SUDS\Cadastral_Maps"  # path to your folder of .shp files
merge = "SWS_INSPIRE"                         # this will be the name of your merged result
directory = os.listdir(path)
count = 0
for filename in directory:
    print "\n" + filename + "\n"
    if ".SHP" in filename.upper() and not ".XML" in filename.upper():
        # On the first pass, create a clone and add the filename column.
        if count == 0:
             # Make a clone (matt wilkie)..
             cmd = 'ogr2ogr ' + path + '\\' + merge + '.shp ' + path + '\\' + filename + ' -where "FID < 0"'
             print "\n" + cmd + "\n"
             os.system(cmd)
             # Add the field (j03lar50n)..
             cmd = 'ogrinfo ' + path + '\\' + merge + '.shp -sql "ALTER TABLE ' + merge + ' ADD COLUMN filename character(50)"'
             print "\n" + cmd + "\n"
             os.system(cmd)
        # Now populate the data (capooti)..
        print "Merging: " + str(filename)
        # You'll need the filename without the .shp extension for the OGR_SQL..
        filenameNoExt = filename.replace(".shp","")
        cmd = 'ogr2ogr -f "esri shapefile" -update -append "' + \
         path + '\\' + merge + '.shp" "' + \
         path + '\\' + filename +'"' \
         ' -sql "SELECT \'' + filename + '\' AS filename, * FROM ' + filenameNoExt + '"'
        # Uncomment this line to spit the ogr2ogr sentence to the terminal..
        print "\n" + cmd + "\n"
        os.system(cmd)
        count += 1
  • Hi Alex, I've had a go at running the code you've posted and it executes without issue. I get a shp file called SWS_INSPIRE in the same folder as the shp files to be merged containing the merged files. Do all of your shp files contain the same type of geometry - are they all points or all polygons...? – ian Dec 14 '18 at 12:50
  • Use os.path.join to join paths and filenames together instead of things like this path + '\\' + filename – BERA Dec 14 '18 at 14:35
8

I'd suggest you rather go with ogr2ogr in a terminal script directly.

In summary (using the syntax from the linked post), to merge all .shp into merged.shp (both in CWD), with the filename (without extension) added as a column, run from within

  • Bash (Linux):

    for file in *.shp
    do
      if [ -f  merged.shp ]
        then
          ogr2ogr -f "ESRI Shapefile" merged.shp $file -update -append -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
        else
          ogr2ogr -f "ESRI Shapefile" merged.shp $file -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
      fi
    done
    
  • CMD (Windows Command Line):

    for %F in (*.shp) do (
      if not exists merged.shp (
        ogr2ogr -f "ESRI Shapefile" merged.shp %F -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
      ) else (
        ogr2ogr -f "ESRI Shapefile" merged.shp %F -update -append -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
      )
    )
    

And to add up to this, there's actually no need to catch the non-existing file case, ogr2ogr will (at least in recent versions) create the file even in -append mode:

  • Bash (Linux):

    for file in *.shp
    do
      ogr2ogr -f "ESRI Shapefile" merged.shp $file -update -append -dialect "SQLite" -sql "SELECT '${file%.*}' AS filename, * FROM ${file%.*}"
    done
    
  • CMD (Windows Command Line):

    for %F in (*.shp) do (
      ogr2ogr -f "ESRI Shapefile" merged.shp %F -update -append -dialect "SQLite" -sql "SELECT '%~nF' AS filename, * FROM %~nF"
    )
    
2

Turns out the path to the ogr2ogr executable wasn't included in my Path environment variable... seems to work now.

1

Your question is tagged with qgis so I'm answering with QGIS, even if you show us only code with ogr2ogr.

In QGIS, you can use the tool Merge vector layers. Two attributes are added automatically, the filename and the filepath.

Then if you want to do the same in python, you can open the history of Processing commands (from the previous step above with Merge vector layer) and copy/paste the python line.

It gives:

processing.run("native:mergevectorlayers", {'LAYERS':['path/to/first/layer','path/to/second/layer'],'CRS':None,'OUTPUT':'memory:'})

It will create a memory layer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.