Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Directory structure is basically as such: county\City\prep\POI\Completed\C1, ...C2, ...etc\

I'm trying to write a script to go through the *\Completed*\POIS.shp and merge them all together, at a country level. So the input would be the country folder as root.

So, to clarify:

county1\City1\prep\POI\Completed\C1\POIS.shp

county1\City1\prep\POI\Completed\C2\POIS.shp

county1\City2\prep\POI\Completed\C1\POIS.shp

etc

I started to write out the walk through the folders, but I figured I should get some guidance first. So now I've got it working so if the merged files are in a specific folder I can easily run the XY coordinates and other selections I need to do. The step just above where I've commented out should be the merge of all POIS.shp files and copied to the directory to run the other tools. Hopefully (but probably not) not confusing.

import arcpy
import os
import sys


# Folder name to run XY script
folder = arcpy.GetParameterAsText(0)
##countryFolder = arcpy.GetParameterAsText(1)
# workspace
arcpy.env.workspace = folder 

# List shapefiles
arcpy.AddWarning("Building List of Shapefiles...")
fcList = arcpy.ListFeatureClasses()

## Merge a Country's completed POI files (Country\*\PDA\Completed_Field_Files\*\POIS.shp\)
##for dirs in os.walk(countryFolder)
##    arcpy.Merge_management("POIS.shp",

# Add coordinates # code is good and works!
try:
    for fc in fcList:
        arcpy.MakeFeatureLayer_management(fc, "temp_" + fc)
        shapeFile = "temp_" + fc 
        # Add XY coor to table
        arcpy.AddXY_management(shapeFile)
        arcpy.AddWarning("Building" + " " + shapeFile + " " + "XY Coordinates Successful")
        # Select only records we want with SQL: Action<>D; Position=Yes,NA; POI_ST_NUM<>blank
        arcpy.AddWarning("Selecting" + " " + shapeFile + " " + "Records...")
        try:
            arcpy.SelectLayerByAttribute_management(shapeFile,"NEW_SELECTION","POI_ST_NUM <> ' '")
        except:
            arcpy.AddWarning("No Blank Street Numbers")
        try:
            arcpy.SelectLayerByAttribute_management(shapeFile,"REMOVE_FROM_SELECTION","ACTION_COD = 'D' OR ACTION_COD = 'FR' OR ACTION_COD = 'AFR'")
        except:
            arcpy.AddWarning("No Action Codes to Remove")
        try:
            arcpy.SelectLayerByAttribute_management(shapeFile,"REMOVE_FROM_SELECTION", "POSITION = 'NO' OR POSITION = ' '")
        except:
            arcpy.AddWarning("No Position Errors to Remove")
        # Copy selected records in feature claseses to new shapefile
        arcpy.AddWarning("Creating New Shapefile...")
        arcpy.CopyFeatures_management(shapeFile, "RWT_XY_" + fc)
    arcpy.AddWarning("***********************")
    arcpy.AddWarning("Huzzah! File/s Complete")
    arcpy.AddWarning("***********************")
except:
    arcpy.AddError("*********************")
    arcpy.AddError("DOH! Adding XY Failed")
    arcpy.AddError("*********************")

Thanks! By the way, this is my 2nd script ever with no previous training or knowledge, so be easy on me. :-D

EDIT: try to make it more clear

I have dozens POIS.shp files I want to merge, but they are all located in different folders according to the structure I listed above. Always in a C1, C2, C3, ..., C16 folder. The C folders are always in Completed_Field_Files folder. The structure may differ a little bit, so I want the tool to crawl and merge all POIS.shp files within the Completed_Field_Fieles\C# folders. There's other POIS.shp files as well, so I ONLY want the ones within those specific folders. I'm basically just merging a bunch of shapefiles that are located in different but specific folders

share|improve this question
4  
Are you using ArcGIS 10.1? If so, I would recommend arcpy.da.Walk over os.walk. –  PolyGeo Mar 14 '13 at 21:43
4  
There are a couple of other questions on this site that should help you with the syntax of walking the folders looking for shapefiles: How to make a GIS inventory? and ArcGIS 10 Python script to search directories and subdirectories for intersects and clip them –  RyanDalton Mar 14 '13 at 23:09
1  
I would get rid of the try/excepts (and the overly verbose messaging). They serve no purpose and will only obscure any actual errors, making debugging more difficult. –  blah238 Mar 14 '13 at 23:48
    
Can you edit your question to clarify step by step what the desired behavior and end result is given a set of inputs and data (file structure)? The question is not clear enough to propose an answer at this point, and the code is not well structured and difficult to follow. You might consider breaking it into functions where each does a small amount of work with a well defined scope and purpose. If you do this you might even be able to figure out the rest on your own, as it will get you thinking about your program's flow of control and design a bit more. –  blah238 Mar 15 '13 at 0:19
1  
When a tool/function fails, it fails for a reason. You want to find that reason. You don't do it by covering it up with a try/except that does nothing (actually, its worse than doing nothing, it lets the program continue in a potentially invalid state). Take out all the exception handlers and run it in a debugger. That way when the error does occur you can see the error message, stack trace, state of all the variables at that time, etc. and hopefully deduce the problem. I would suggest using an IDE such as PyScripter or Eclipse with PyDev. –  blah238 Mar 15 '13 at 20:03
show 2 more comments

1 Answer

up vote 4 down vote accepted

Something like this (untested) should put you on the right track:

import os
import arcpy
import fnmatch

def listCountyPOIs(county_folder):
    pois = []
    for dirpath, dirnames, filenames in os.walk(county_folder):
        for filename in filenames:
            fullpath = os.path.join(dirpath, filename)
            if fnmatch.fnmatch(fullpath, r"*\Completed_Field_Files\C*\POIS.shp"):
                pois.append(fullpath)
    return pois

if __name__ == "__main__":
    countyFolder = arcpy.GetParameterAsText(0)
    outputShapefile = arcpy.GetParameterAsText(1)
    countyPOIs = listCountyPOIs(countyFolder)
    arcpy.Merge_management(countyPOIs, outputShapefile)

A more generic version of this function is given below. This is a generator function.

def matchFiles(root_dir, pattern):
    for dirpath, dirnames, filenames in os.walk(root_dir):
        for filename in filenames:
            fullpath = os.path.join(dirpath, filename)
            if fnmatch.fnmatch(fullpath, pattern):
                yield fullpath

Usage (using the OP's example):

pattern = r"*\Completed_Field_Files\C*\POIS.shp"
countyPOIs = list(matchFiles(countyFolder, pattern))
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.