-1

I have about 10 standard CSVs that are occasionally used to update (overwrite) their related shapefiles in Arc. The CSVs have different quantity and names of their fields. Their fields can also be a variety of data types (e.g. C, D, F, etc.)

I'm using Arc 10.2.2.

Can anyone suggest the simplest sequence of arcpy functions to use in a script that will convert a csv to a feature class (and overwrite if pre-existing)?

I'm not asking for a complete script, just the appropriate arcpy functions in the proper order.

For example (although not applicable to my task) a series of arcpy functions like: arcpy.ExcelToTable_conversion("data.xls", "outgdb.gdb", "Sheet1")

The script needs to be able to handle a wide variety of CSVs. However, you may assume the CSV fields exactly match those in the target Feature Class to be overwritten.

closed as off-topic by PolyGeo May 3 '18 at 7:08

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking help to debug/write/improve code must include the desired behavior, a specific problem or error and the shortest code necessary to reproduce it in the question itself. Providing a clear problem statement and evidence of a code attempt will help others to help you. See: How to create a Minimal, Complete, and Verifiable example." – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • Are the geometry (X, Y and optional Z) the same for all feature classes? When importing from CSV I usually import into Microsoft Access to cement the field types and clean out any nasties before importing into a feature class but if you want to go directly I suggest to create a feature class from a template resources.arcgis.com/en/help/main/10.2/index.html#//…, set the spatial reference and then read the CSV rows as text and create a geometry point from the identified location columns then insert using a cursor. Does that help? – Michael Stimson May 3 '18 at 1:01
  • All Feature classes are point feature classes (so just X and Y). That is what you mean? – Waterman May 3 '18 at 1:20
  • I take it from your answer that my hope for a series of pre-existing arcpy functions is a pipe dream. – Waterman May 3 '18 at 1:23
  • It sounds more difficult than it is.. What you need to be able to determine at run time is the name of the fields in the CSV that contains the geometry (X coord, Y coord and optional Z height); are they all called 'lat,lon', 'easting,northing', 'x,y'? everything else is fairly straightforward; the suggestion to use Access stems from having some fairly dodgy CSV data supplied, it's a cleansing process that fixes up a lot of errors that can halt a process, if you're confident that your CSV data is clean then you can import without this step. – Michael Stimson May 3 '18 at 2:01
  • 2
    Unfortunately, identifying the functions to be called and their order is close enough to coding a solution that this is more likely than not to be judged either too broad or in need of a coding attempt on your part. Generic solutions require a great deal more code than fixed ones. And, as is not uncommon, there are a number of ways to reach this particular destination. – Vince May 3 '18 at 2:21
2

Have a go at this:

import os, sys, arcpy

InCSV  = sys.argv[1] # your CSV file
OutShp = sys.argv[2] # the existing file to replace (required)

# Populate the expected input X and Y field names in upper case
XCSV   = 'EASTING' 
YCSV   = 'NORTHING'

# need to separate the folder and the name for create feature class
OutDir  = os.path.dirname(OutShp)  # the folder
OutName = os.path.basename(OutShp) # the file

# change the workspace, now we don't need to give the full path to the template shapefile
arcpy.env.workspace = OutDir
try:
    arcpy.Rename_management(OutName,'template.shp')
except:
    arcpy.AddError("Unable to rename output shapefile to template.shp")
    sys.exit(-1) # stop here, the file could be locked or template.shp already exists

# http://resources.arcgis.com/en/help/main/10.2/index.html#/Dataset_properties/03q30000008p000000/
# get the spatial reference of the existing template shapefile
Desc = arcpy.Describe('template.shp')
SR   = Desc.spatialReference

# create a new feature class with the template of the old feature class
arcpy.CreateFeatureDataset_management(OutDir,OutName,"POINT",'template.shp',spatial_reference=SR)

FieldMatch = {} # a dictionary to match the field index to the csv index
ShpFields  = [] # the list of fields in the shape file that match
X_Index    = -1 # the index of the location fields
Y_Index    = -1 # default to -1 to test that the fields are set

# open up the CSV as a text file (read only) using with statement, this
# will close the file when we reach the end of the block.
with open(InCSV,'r') as InCSVfile:
    HeaderLine  = InCSVfile.readline().upper() # read the header line and convert to upper case
    HeaderSplit = HeaderLine.split(',')        # break the header line into a list of strings

    # match up the fields from CSV to SHP.. be careful though, the field type is not being tested
    for ThisF in arcpy.ListFields(OutName):
        if ThisF.name.upper() in HeaderSplit:
            # the field name is found, add to the list for the cursor
            ShpFields.append(ThisF.name)
            # add the index to the fieldmatch dict to match the shp field to the csv index
            FieldMatch[len(ShpFields)-1]=HeaderSplit.index(ThisF.name.upper())

    # locate the X and Y index in the CSV
    if XCSV in HeaderSplit:
        X_Index = HeaderSplit.index(XCSV)
    if YCSV in HeaderSplit:
        Y_Index = HeaderSplit.index(YCSV)

    if X_Index == -1 or Y_Index == -1:
        # can't locate x or y field, bail out but first rename 
        # the output shape back to what it was.
        arcpy.AddError("FAIL: location fields not found")
        arcpy.Rename_management('template.shp',OutName)
        sys.exit(-2)

    ShpFields.append('SHAPE@')
    KeyRange = range(len(ShpFields))

    # open an insert cursor on the output shape file
    with arcpy.da.InsertCursor(OutName,ShpFields) as ICur:
        # read the rest of the in CSV file line-by-line
        for ThisLine in InCSVfile:
            ThisSplit = ThisLine.split(',')
            ThisRow   = [] # a new empty list to make a tuple from

            # populate the fields that have matched
            for ThisKey in KeyRange:
                # get the value from the CSV line matching each key
                ThisValue = ThisSplit[FieldMatch[ThisKey]]
                # you might need to do something here to convert strings to numbers
                ThisRow.append(ThisValue)

            # make a new point using the values in the csv line at X_Index and Y_Index
            ThisPnt = arcpy.Point(float(ThisSplit[X_Index]),float(ThisSplit[Y_Index]))
            ThisRow.append(ThisPnt)

            # insert the feature
            ICur.insertRow(tuple(ThisRow))

# clean up old template
arcpy.Delete_management('template.shp')

It's a bit long winded but is as generic as I can make it.. there's a few steps:

  1. Rename the existing file to be overwritten to 'template.shp', so that it's still there but not conflicting.
  2. Create a new feature class using 'template.shp' to define the fields and copying the existing template spatial reference.
  3. Open up the text file and read the header row to get the defined column names which get matched up to existing field names using a dictionary to index the field list to the CSV column number.
  4. Locate the X and Y columns in the CSV - fail if this doesn't work.
  5. Start up an insert cursor on the new, empty shapefile.
  6. Read the rest of the CSV file after the header row which was already read.
  7. Copy the CSV values into a new list to populate the feature, these will be in the same order as the fields in the insert cursor.
  8. Create a new point using the X and Y values (as float), appending the point to the list.
  9. Insert the row as a tuple.
  10. When complete remove 'template.shp' so that it doesn't conflict later.

Not the answer you're looking for? Browse other questions tagged or ask your own question.