UnicodeEncodeError for searchCursor

Question

import arcpy
import csv
import glob, os
os.chdir("C:\\path\\folder")
fieldsource = "C:\\path\\sampleshp.shp"
fields = [r.name for r in arcpy.ListFields(fieldsource)]
csv_out = "C:\\path\\table.csv"
with open(csv_out, "wb") as f:
    wr = csv.writer(f)
    wr.writerow(fields)
    for file in glob.glob("*.shp"):
        in_shp = "C:\\path\\folder\\" + file
        with arcpy.da.SearchCursor(in_shp, fields) as cursor:
            for row in cursor:
                wr.writerow(row)

I have few thousand shapefiles with only a few rows in them. I want to export their table to a single csv file. The script works but it seems I have a character in some of the files that python(2.7) doesn't like:

Traceback (most recent call last): File "C:\path\folder\table.py", line 20, in wr.writerow(row) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

How can I exclude/change the character that creates the issue?

Answer 1

The reason you get this error is because the csv module cannot handle unicode strings that you are trying to write as ASCII.

There are two approaches:

1. Use the .encode method on all strings/unicodes you have in a tuple that you get from da.SearchCursor):

Code:

import arcpy
import csv

fc = r'C:\GIS\Temp\ArcGISHomeFolder\Default.gdb\HomeAddresses'
fields = [r.name for r in arcpy.ListFields(fc) if r.required == False]

with open('addresses.txt', "wb") as f:
    wr = csv.writer(f)
    wr.writerow(fields)
    with arcpy.da.SearchCursor(fc, fields, 'OBJECTID < 10') as cursor:
        for row in cursor:
            row_updated = [i.encode('utf-8') if isinstance(i, (str,unicode)) else i for i in row]
            wr.writerow(row_updated)

2. Use Python module that can do that for you, such as unicodecsv.

Code:

import arcpy
import unicodecsv

fc = r'C:\GIS\Temp\ArcGISHomeFolder\Default.gdb\HomeAddresses'
fields = [r.name for r in arcpy.ListFields(fc) if r.required == False]

with open('addresses_unicodecsv.txt', "wb") as f:
    wr = unicodecsv.writer(f)
    wr.writerow(fields)
    with arcpy.da.SearchCursor(fc, fields, 'OBJECTID < 10') as cursor:
        for row in cursor:            
            wr.writerow(row)