# Why is an extra field necessary when creating point shapefile from csv files in Python?

I have a Python script that creates a point shapefile from a csv file within a Python script. This script gives me a ValueError for the longitude field (LON) if there isn't an extra nonsensical field to the right of it. Below is the error and how the csv file has been formatted to remove this error:

    lonValueIndex = valueList.index("LON")
ValueError: list.index(x): x not in list

DATE       LAT      LON          TEST
2/19/2011  34.27531  -118.21071    a
2/19/2011  34.18069  -118.34079    b
2/19/2011  34.0215   -118.21857    c


Do you see that Test field with values of a, b, and c? If I don't put a field to the right of the LON field I get the ValueError. Below is part of my script. Does anyone know why that extra field is necessary? Thank you.

import arcpy, csv
arcpy.env.overwriteOutput = True

#Set variables
arcpy.env.workspace = "C:\\GIS\\StackEx\\"
outFolder = arcpy.env.workspace
pointFC = "art2.shp"
coordSys = "C:\\Program Files\\ArcGIS\\Desktop10.0\\Coordinate Systems" + \
"\\Geographic Coordinate Systems\\World\\WGS 1984.prj"
csvFile = "C:\\GIS\\StackEx\\chicken.csv"
fieldName = "DATE1"

arcpy.CreateFeatureclass_management(outFolder, pointFC, "POINT", "", "", "", coordSys)

gpsTrack = open(csvFile, "r")

print valueList
latValueIndex = valueList.index("LAT")
lonValueIndex = valueList.index("LON")
dateValueIndex = valueList.index("DATE")

# Read each line in csv file
cursor = arcpy.InsertCursor(pointFC)

segmentedPoint = point.split(",")
# Get the lat/lon values of the current reading
latValue = segmentedPoint[latValueIndex]
lonValue = segmentedPoint[lonValueIndex]
dateValue = segmentedPoint[dateValueIndex]
vertex = arcpy.CreateObject("Point")
vertex.X = lonValue
vertex.Y = latValue
feature = cursor.newRow()
feature.shape = vertex
feature.DATE1 = dateValue
cursor.insertRow(feature)

del cursor

-
Post the entire script for us to be able to give better answers. You might also consider using the csv module to read the file. –  blah238 Dec 6 '11 at 3:24
Hello, I added the entire script, as you can see I imported the csv module but I haven't used it, yet. Thank you. –  Patty Jula Dec 6 '11 at 4:04
Can you post a few lines from the CSV file including the header as plain text? Also the line gpsTrack = (csvFile, "r") appears to be an editing typo, as that would not have the desired result :) –  blah238 Dec 6 '11 at 4:48
Plain text of CSV file added. Thank you. –  Patty Jula Dec 6 '11 at 4:59
Well, without having the file in hand, and based on your workaround of adding a dummy field onto the end, I suspect the culprit is a newline character stuck on the end of the last value in the list returned by split(). This is one of the things the csv module was designed to account for. So when you use index() to look for LON, it's not finding it because what you should really be looking for is LON\r\n or similar, which is of course unintuitive. –  blah238 Dec 6 '11 at 5:26

When you split headerLine by the commas, the last entry "LON" contains a newline character:

print headerLine.split(",")
['DATE', 'LAT', 'LON\n']


This is why adding a new column after LON causes the script to work, since the newline character is applied to the TEST field.

Try using STRIP to remove the newline character:

print headerLine.strip().split(",")
['DATE', 'LAT', 'LON']

-
Exactly! 10char –  blah238 Dec 6 '11 at 5:34
@blah238 I just learned something new (not about GIS - the 10char thing). Thanks! –  Stephen Lead Dec 6 '11 at 5:37
Ah, that was it. I changed the valueList definition to remove the '\n'. My script has been updated above. Thank you! –  Patty Jula Dec 6 '11 at 16:19

Here is an alternative workflow using geoprocessing tools that may be easier:

1. Use Make XY Event Layer (Data Management) to create a temporary point layer from your CSV file

2. Use Copy Features (Data Management) or Feature Class To Feature Class (Conversion) to make a permanent feature class or shapefile from the temporary point layer.

-
That looks like an excellent solution... probably a method I would switch to in the future. But I'm really after finding out why the extra field is necessary. –  Patty Jula Dec 6 '11 at 4:51

If you are willing to use OGR/GDAL, your problems can be solved really easily. If you can export your data to be comma separated (or tab separated), and put it into a file called test.csv. Then you can create an XML file that will define where the lat/long columns are to be read from:

<OGRVRTDataSource>
<OGRVRTLayer name="test">
<SrcDataSource>test.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<LayerSRS>WGS84</LayerSRS>
<GeometryField encoding="PointFromColumns" x="LAT" y="LON"/>
</OGRVRTLayer>
</OGRVRTDataSource>


Then, to get your shapefile, use ogr2ogr:

$ogr2ogr -f "ESRI Shapefile" /tmp/stuff.shp /tmp/test.vrt$ ogrinfo -al -so stuff.shp
INFO: Open of stuff.shp'
using driver ESRI Shapefile' successful.

Layer name: stuff
Geometry: Point
Feature Count: 3
Extent: (34.021500, -118.340790) - (34.275310, -118.210710)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
DATUM["WGS_1984",
SPHEROID["WGS_1984",6378137,298.257223563]],
PRIMEM["Greenwich",0],
UNIT["Degree",0.017453292519943295]]
DATE: String (80.0)
LAT: String (80.0)
LON: String (80.0)
TEST: String (80.0)


There's more documentation here.

-