2

I have a script that I've used to import information in a CSV to a shapefile.

What I'd like to do is make it so that this script will run on all CSVs in a specific folder location, creating multiple shapefiles.

This is a bit beyond my current programming skills. Can someone point me in the right direction? The current script that imports from a single CSV is below.

import arcpy
import csv
import os
print "arcpy imported"

"""
# Author: John K. Tran
# Contact: jtran20@masonlive.gmu.edu

Input CSV can look like:

Lat,Lon,First Name,Last Name
12.34,56.78,Joe,Smith
0.98,7.65,Jane,Doe
65.43,43.21,Bob,Sagat

Make sure 'Lat' and 'Lon' fields are the 1st and 2nd column in the CSV 
respectively.

"""


incsv = arcpy.GetParameterAsText(0) # Change this to the path of your CSV file.
outfc = arcpy.GetParameterAsText(1) + "\\" + arcpy.GetParameterAsText(2) + ".shp" # Change this to the path of your output FC.

spatialref = arcpy.SpatialReference(4326) # Create the spatial reference object as WGS84. Can modify if desired.

if not arcpy.Exists(outfc): # Create the output feature class if needed. (if this is not an existing path, create it)
    arcpy.CreateFeatureclass_management(os.path.dirname(outfc), 
os.path.basename(outfc), "POINT", None, None, None, spatialref)

csv.register_dialect("xls", delimiter=",", lineterminator="\n") # Register the dialect for native CSV syntax in Microsoft Excelself.
f = open(incsv, "r")
reader = csv.reader(f, dialect = "xls")

headers = reader.next() # Read the first line as the header names.

for header in headers[2:]: # Add fields for remaining columns if needed. Default is TEXT field.
    arcpy.AddField_management(outfc, header, "TEXT")

cursor = arcpy.da.InsertCursor(outfc, ['SHAPE@XY'] + headers[2:]) # Create 
InsertCursor.
count = 0
for row in reader:
    if count % 1000 == 0:
        print "processing row {0}".format(count)
        Ycoord = row[0] # Make sure 'Lat' is in the 1st column.
        Xcoord = row[1] # Make sure 'Lon' is in the 2nd column.
        newrow = [(float(Xcoord), float(Ycoord))] + row[2:]
        cursor.insertRow(newrow) # Insert point in FC for each row in CSV.
        count += 1

del cursor
f.close()
3

Expanding on my comment:

import arcpy
import csv
import os
print "arcpy imported"

"""
# Author: John K. Tran
# Contact: jtran20@masonlive.gmu.edu

Input CSV can look like:

Lat,Lon,First Name,Last Name
12.34,56.78,Joe,Smith
0.98,7.65,Jane,Doe
65.43,43.21,Bob,Sagat

Make sure 'Lat' and 'Lon' fields are the 1st and 2nd column in the CSV 
respectively.

"""


CSVfolder = arcpy.GetParameterAsText(0) # Change this to the path of your CSV files.
SHPfolder = arcpy.GetParameterAsText(1) # Change this to the path of your output FC.

spatialref = arcpy.SpatialReference(4326) # Create the spatial reference object as WGS84. Can modify if desired.

arcpy.env.overwriteOutput = True # set overwrite, if the shapefile already exists it will be created

for incsv in os.listdir(CSVfolder): # loop through ALL files in the input folder
    if os.path.splitext(incsv)[1].lower() == '.csv': # check the file is a CSV file
        outfc = os.path.join(SHPfolder,os.path.splitext(incsv)[0] + '.shp')
        # create the new shapefile here
        arcpy.CreateFeatureclass_management(SHPfolder,os.path.basename(outfc), "POINT", None, None, None, spatialref)
        csv.register_dialect("xls", delimiter=",", lineterminator="\n") # Register the dialect for native CSV syntax in Microsoft Excelself.
        with open(os.path.join(CSVfolder,incsv), "r") as f:
          reader = csv.reader(f, dialect = "xls")

          headers = reader.next() # Read the first line as the header names.

          for header in headers[2:]: # Add fields for remaining columns if needed. Default is TEXT field.
                  arcpy.AddField_management(outfc, header, "TEXT")
          # using a with block to create and manage the insert cursor.. does not work with arcpy.InsertCursor
          # ensures that the resources are properly released at the end.
          with arcpy.da.InsertCursor(outfc, ['SHAPE@XY'] + headers[2:]) as cursor:
              count = 0
              for row in reader:
                      if count % 1000 == 0:
                              print "processing row {0}".format(count)
                              Ycoord = row[0] # Make sure 'Lat' is in the 1st column.
                              Xcoord = row[1] # Make sure 'Lon' is in the 2nd column.
                              newrow = [(float(Xcoord), float(Ycoord))] + row[2:]
                              cursor.insertRow(newrow) # Insert point in FC for each row in CSV.
                              count += 1

        # del cursor # no longer needed with the with block.
        # f.close() # no longer needed with the with block.

I have changed the arguments to the folder the CSV files are in and the folder you want the shapefiles in. The shape name will be the same as the CSV name with the extension changed to .shp.

I am using os.listdir() but arcpy.ListFiles() might work better as you can supply a wildcard (*.csv). Then using os.path.splitext(filename) will break the file name into a tuple of (name, extension), I use the 2nd element (index 1 as indexing is 0 based) converted to lower case to check the file is a CSV before progressing to the next line.

  • 1
    You could also use the glob stdlib to do all of that in one line. csv_paths = glob.glob(os.path.join(csv_folder, '*.csv')) – mikewatt Nov 6 '18 at 0:58
  • @gberard that's a good suggestion, there are indeed many ways to achieve this but as the original question was already importing os I would stick to the libs the user is currently using - the OP is a beginner I didn't want to confuse the topic by introducing more advanced libs. – Michael Stimson Nov 6 '18 at 1:11
  • Hi both, thank you for your suggestions. @MichaelStimson, I ran the code that you wrote and came up with the following error: Traceback (most recent call last): File "U:\skelley\Python\Scripts\FWDimport2.py", line 36, in <module> f = open(incsv, "r") IOError: [Errno 2] No such file or directory: u'FWD_Halsey_WGS1984.csv' Failed to execute (FWDImport). – SarahKelley Nov 6 '18 at 17:16
  • Note - I made one minor change, removed this piece (as it was for naming a single file and was no longer needed): + "\\" + arcpy.GetParameterAsText(2) + ".shp" Apologies for the formatting - I can't seem to get my comments to display the code properly. Not sure what I'm doing wrong. – SarahKelley Nov 6 '18 at 17:21
  • Doh! you're right, there is no need for the extra 3rd argument. I should have noticed that; I'll fix it straight away. Comments (and chat) don't display code correctly, especially python where indentation is vital. Does the code work for you now? – Michael Stimson Nov 7 '18 at 0:53
2

You can use MakeXYEventLayer instead of insertcursor. No need to add fields etc.:

import arcpy
arcpy.env.overwriteOutput = True
from os import listdir
from os.path import isfile, join, basename

csvpath = r'C:\Test\Testmapp'
outshapefolder = r'C:\Test\Outputshapes'

files = [join(csvpath,f) for f in listdir(csvpath) if isfile(join(csvpath, f))]

for f in files:
    arcpy.MakeTableView_management(in_table=f, out_view='tempview')
    arcpy.MakeXYEventLayer_management(table='tempview', in_x_field='Lat', in_y_field='Lon', 
                                     out_layer='tempevent', spatial_reference=4326)
    arcpy.CopyFeatures_management(in_features='tempevent', out_feature_class=join(outshapefolder,basename(f).replace('.csv','.shp')))
  • Thanks! This looks a lot like my first attempt. I was thinking of going back to this approach - the only reason I used the approach in my original post was because I found it in a thread that discussed which method would import CSV data faster - the consensus was that the geoprocessing tools were slower. However, it might be worth trying both ways and comparing the speed myself. Appreciate the response! – SarahKelley Nov 7 '18 at 21:57
  • In my experience the geoprocessing tools are faster than anything done solely in python. There is some overhead converting python objects to C objects and a lot more goes on in the background of python interpreting instructions whereas the geoprocessing tools are already compiled and linked and have almost no overhead. The problem I have with importing CSV files using this method is that the ArcGIS handling of CSV files, especially large ones, is dodgy at best. I suggest using MS Office (or similar) to import the CSV into a personal geodatabase table for reliability before using this method. – Michael Stimson Nov 7 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.