Why is an extra field necessary when creating point shapefile from csv files in Python?

Question

I have a Python script that creates a point shapefile from a csv file within a Python script. This script gives me a ValueError for the longitude field (LON) if there isn't an extra nonsensical field to the right of it. Below is the error and how the csv file has been formatted to remove this error:

    lonValueIndex = valueList.index("LON")
    ValueError: list.index(x): x not in list

DATE       LAT      LON          TEST
2/19/2011  34.27531  -118.21071    a
2/19/2011  34.18069  -118.34079    b
2/19/2011  34.0215   -118.21857    c

Do you see that Test field with values of a, b, and c? If I don't put a field to the right of the LON field I get the ValueError. Below is part of my script. Does anyone know why that extra field is necessary? Thank you.

import arcpy, csv
arcpy.env.overwriteOutput = True

#Set variables
arcpy.env.workspace = "C:\\GIS\\StackEx\\"
outFolder = arcpy.env.workspace
pointFC = "art2.shp"
coordSys = "C:\\Program Files\\ArcGIS\\Desktop10.0\\Coordinate Systems" + \
           "\\Geographic Coordinate Systems\\World\\WGS 1984.prj"
csvFile = "C:\\GIS\\StackEx\\chicken.csv"
fieldName = "DATE1"

#Create shapefile and add field
arcpy.CreateFeatureclass_management(outFolder, pointFC, "POINT", "", "", "", coordSys)
arcpy.AddField_management(pointFC, fieldName, "TEXT","","", 10)

gpsTrack = open(csvFile, "r")

headerLine = gpsTrack.readline()
#print headerLine
#I updated valueList to remove the '\n'
valueList = headerLine.strip().split(",")
print valueList
latValueIndex = valueList.index("LAT")
lonValueIndex = valueList.index("LON")
dateValueIndex = valueList.index("DATE")

# Read each line in csv file
cursor = arcpy.InsertCursor(pointFC)
for point in gpsTrack.readlines():

   segmentedPoint = point.split(",")
   # Get the lat/lon values of the current reading                    
   latValue = segmentedPoint[latValueIndex]
   lonValue = segmentedPoint[lonValueIndex]
   dateValue = segmentedPoint[dateValueIndex]
   vertex = arcpy.CreateObject("Point")
   vertex.X = lonValue
   vertex.Y = latValue
   feature = cursor.newRow()
   feature.shape = vertex
   feature.DATE1 = dateValue
   cursor.insertRow(feature)

del cursor

Answer 1

When you split headerLine by the commas, the last entry "LON" contains a newline character:

print headerLine.split(",")
['DATE', 'LAT', 'LON\n']

This is why adding a new column after LON causes the script to work, since the newline character is applied to the TEST field.

Try using STRIP to remove the newline character:

print headerLine.strip().split(",")
['DATE', 'LAT', 'LON']

Steve

Answer 2

Here is an alternative workflow using geoprocessing tools that may be easier:

1. Use Make XY Event Layer (Data Management) to create a temporary point layer from your CSV file

2. Use Copy Features (Data Management) or Feature Class To Feature Class (Conversion) to make a permanent feature class or shapefile from the temporary point layer.

Answer 3

If you are willing to use OGR/GDAL, your problems can be solved really easily. If you can export your data to be comma separated (or tab separated), and put it into a file called test.csv. Then you can create an XML file that will define where the lat/long columns are to be read from:

<OGRVRTDataSource>
  <OGRVRTLayer name="test">
    <SrcDataSource>test.csv</SrcDataSource>
    <GeometryType>wkbPoint</GeometryType>
    <LayerSRS>WGS84</LayerSRS>
    <GeometryField encoding="PointFromColumns" x="LAT" y="LON"/>
  </OGRVRTLayer>
</OGRVRTDataSource>

Then, to get your shapefile, use ogr2ogr:

$ ogr2ogr -f "ESRI Shapefile"  /tmp/stuff.shp /tmp/test.vrt 
$ ogrinfo -al -so stuff.shp
INFO: Open of `stuff.shp'
using driver `ESRI Shapefile' successful.

Layer name: stuff
Geometry: Point
Feature Count: 3
Extent: (34.021500, -118.340790) - (34.275310, -118.210710)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
   DATUM["WGS_1984",
     SPHEROID["WGS_1984",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
UNIT["Degree",0.017453292519943295]]
DATE: String (80.0)
LAT: String (80.0)
LON: String (80.0)
TEST: String (80.0)

There's more documentation here.