1

I have this table with one row

timestamp_pretty   3.1.2014 9:13
timestamp          1,38874E+12
msgid              3
targetType         A
mmsi               205533000
lat                53.4346
long               14.580546
posacc             0
sog                0
cog                319.5
shipType           CARGO
dimBow             68
draught            4
dimPort            6
dimStarboard       5
dimStern           12
month              1
week               1
imo                8404446
country            Belgium
name               FAST JULIA

I want to make a point feature class from arcpy using insert cursor:

# Read the csv
csv.register_dialect("xls", lineterminator="\n")
f = open(incsv, "r")
reader = csv.reader(f, dialect = "xls")

# Add fields to fc
desc = arcpy.Describe(incsv)
for field in desc.fields:
    arcpy.AddField_management(outfc,field.name,field.type)

# the fieldnames
fields = desc.fields
fieldnames = [field.name for field in fields]

 # Create InsertCursor.
cursor = arcpy.da.InsertCursor(outfc, ['SHAPE@XY'] + fieldnames)
count = 0
next(reader, None) # skip the header
for row in reader:
    if count % 10000 == 0:
        print "processing row {0}".format(count) + " of " + table
    Ycoord = row[5]
    Xcoord = row[6]
    newrow = [(float(Xcoord), float(Ycoord))] + row[0:]
    cursor.insertRow([newrow])
    count += 1
del cursor
f.close()

But I get this error:

line 130, in <module>
    cursor.insertRow([newrow])
TypeError: sequence size must match size of the row

I've been through SE similar answers and made many tests (days) but to no avail.

****EDIT****

If I print the result of newrow and row[0:] like this:

newrow = [(float(Xcoord), float(Ycoord))] + row[0:]
print "new row: "+str(newrow)
print "row[0:]: "+str(row[0:])

*EDIT 2 * name and type use for create feature class

[u'timestamp_pretty', u'timestamp', u'msgid', u'targetType', u'mmsi', u'lat', u'long', u'lat_D', u'long_D', u'posacc', u'sog', u'cog', u'shipType', u'dimBow', u'draught', u'dimPort', u'dimStarboard', u'dimStern', u'month', u'week', u'imo', u'country', u'name']
[u'Date', u'Double', u'Integer', u'String', u'Integer', u'String', u'String', u'Double', u'Double', u'Integer', u'String', u'String', u'String', u'Integer', u'String', u'Integer', u'Integer', u'Integer', u'Integer', u'Integer', u'Integer', u'String', u'String']

I get this result:

new row: [(14.580546, 53.4346), '03/01/2014 09:13:26', '1388740406000', '3', 'A', '205533000', '53.4346', '14.580546', '0', '0', '319.5', 'CARGO', '68', '4', '6', '5', '12', '01', '01', '8404446', 'Belgium', 'FAST JULIA']
row[0:]: ['03/01/2014 09:13:26', '1388740406000', '3', 'A', '205533000', '53.4346', '14.580546', '0', '0', '319.5', 'CARGO', '68', '4', '6', '5', '12', '01', '01', '8404446', 'Belgium', 'FAST JULIA']

I now, newrow has 22 fields (counting the coordinates in the beginning) and row[0:] has 21. Is that the error? If so why did it work in the original script I got from @John?

  • 1
    Test with tuple no list of values. > newrow = ((x,y), *row) > cursor. insertRow(newrow). You can test len fieldnames equals len row[0:] – GeoStoneMarten Nov 26 '15 at 9:39
  • Please copy the python object content of row[0:] and fieldnames in your post. I am sur that is a synthax error of your row or your fieldnames structure (message error is clear). Add break before line in cursor.insertRow... and get value in debug mode or interactive interface. Thanks – GeoStoneMarten Nov 26 '15 at 11:29
  • note: if you create a feature class with has a shapefile, the process add a colomn named ID. First you need check if column is created, Second check if all source colomn file find a correspondance (import collection). If outputfile is shape, column name is truncated to 10 charaters. You can catch name and set it on collection object and use it later on a cursor with list of fields. Note: if cursor dont't find name, cursor ignore the fields and you can't add value. "C'est la vie" ;-) – GeoStoneMarten Nov 26 '15 at 12:36
  • 1
    ok for row but convert it with tuple(newrow) . Next check (print) value of fieldnames (because this is fields of csv) and create same for fieldnamesfc = [field.name for field in arcpy.Describe(outfc).fields]. I think problem is here. – GeoStoneMarten Nov 27 '15 at 11:36
  • In fieldnamesfc I get two new fields: [u'OBJECTID', u'Shape', ...]. – Manuel Frias Nov 27 '15 at 12:04
1

All this code is overkill, you can convert your entire table into a layer using the Make XY Event Layer tool, in one line of code! See the code sample on the help page of the tool.

If you need the dataset to be permanent then you can use the Copy Features tool, another line of code!

These could be strung together within modelbuilder or called from within a python script.

  • 1
    My tables are about 2 GB. According to previous tests, Copy Features tool can take (many) hours (if it doesn't crash). A while ago I was looking for a fast method to convert tables to FC and @John kindly provided a script (gis.stackexchange.com/questions/160002/…) that was really fast. I am trying to make it more flexible – Manuel Frias Nov 26 '15 at 10:55
  • 1
    Make XY Event is in memory process and can crach with big dataset (depend memory machin and other process running). @Hornbydd Your response make sense as a possible alternative. But this is not a response to this question. – GeoStoneMarten Nov 26 '15 at 13:22
  • Thanks for the tips @GeoStoneMarten, A was also testing the inmemory but it didn't quite work either. I guess it's related to the size of the file. – Manuel Frias Nov 27 '15 at 11:21
  • I don't think so. I work with big csv table and don't have any problem. Problems are data contains or len of rows – GeoStoneMarten Nov 27 '15 at 11:39
1
  1. first simple solution: force text type

    # Read the csv
    csv.register_dialect("xls", lineterminator="\n")
    f = open(incsv, "r")
    reader = csv.reader(f, dialect = "xls")
    
    # Add fields to fc
    desc = arcpy.Describe(incsv)
    for field in desc.fields:
        arcpy.AddField_management(outfc,field.name,"TEXT")  #force text format
    
    # the fieldnames
    fields = desc.fields
    fieldnames = [field.name for field in fields]
    
     # Create InsertCursor.
    cursor = arcpy.da.InsertCursor(outfc, ['SHAPE@XY'] + fieldnames)
    count = 0
    # TODO: create fieldrow positon detector function with ignore case
    # check_x = ['x','lng', 'long', 'longitude'] and other use
    # check_y = ['y','lat', 'latitude'] and other use
    next(reader, None) # skip the header
    
    for row in reader:
        if count % 10000 == 0:
            print "processing row {0}".format(count) + " of " + table
    
        Ycoord = row[5] # set row with position detector
        Xcoord = row[6] # set row with position detector
        newrow = [(float(Xcoord), float(Ycoord))] + row[0:]        
        cursor.insertRow(newrow)  # change here don't add double list and prefer pass tuple : tuple(newrow) if row is <list>
        count += 1
    del cursor
    f.close()
    
  • That's the way... almost. I need a feature class with the original types. At least for some of the fields as the date (timestamp_pretty) and other field called mmsi. The tables are not in a GDB so a ListTables won't work. I guess I need to do it via csv sniffer? – Manuel Frias Nov 27 '15 at 15:16
  • @Manuel Frias Sniffer can help you to detect format of the csv files. The delimiter characters may vary between files. Sniffer class, we can handle in this case and sniff method generate a dialect. see that discussion – GeoStoneMarten Nov 27 '15 at 15:22
  • @ManuelFrias The Sniffer can detect a good part but not all. If you know format of csv, you can create an intermediate parser to fix row data with a good format specified in your fc. The big problem is datetime format row['DateTime'] = datetime.strptime('Jun 8 2015 1:33PM', '%b %d %Y %I:%M%p') – GeoStoneMarten Nov 27 '15 at 15:40
  • @ManuelFrias I test format datetime for import in table with arcpy and cur.insertRow(((3.8, 43.8), "12/12/2015 15:52:24")) return same result to cur.insertRow(((3.8, 43.8), datetime(2015, 12, 12, 15, 52, 24, 0))). For work, date(in str format) must be stored in the local system format (for me %d/%m/%y %H:%M:%S") and time is optional i think arcpy can detect local parameters and for exemple Make XY Event want x and y with comma separator in French ArcGIS Sofware and ; is default csv separator in French export XLS – GeoStoneMarten Nov 27 '15 at 16:10
  • I found a post it can interest you because you can set a parser in reader with panda look at this – GeoStoneMarten Nov 28 '15 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.