Fastest methods for modifying attribute tables with Python?

Question

A while ago, I wrote a quick Python function for converting an attribute table to a python dictionary, where the key is taken from a user-specified unique ID field (typically the OID field). Additionally, by default all fields are copied to the dictionary, but I've included a parameter allowing for just a subset to be specified.

def make_attribute_dict(fc, key_field, attr_list=['*']):
    dict = {}
    fc_field_objects = arcpy.ListFields(fc)
    fc_fields = [field.name for field in fc_field_objects if field.type != 'Geometry']
    if attr_list == ['*']:
        valid_fields = fc_fields
    else:
        valid_fields = [field for field in attr_list if field in fc_fields]
    if key_field not in valid_fields:
        cursor_fields = valid_fields + [key_field]
    else:
        cursor_fields = valid_fields
    with arcpy.da.SearchCursor(fc, cursor_fields) as cursor:
        for row in cursor:
            key = row[cursor_fields.index(key_field)]
            subdict = {}
            for field in valid_fields:
                subdict[field] = row[cursor_fields.index(field)]
            dict[key] = subdict
            del subdict
    return dict

This works great for relatively small datasets, but I just ran it on a table containing about 750,000 rows and 15 fields -- around 100MB in a file geodatabase. On these, the function runs much slower than I would have expected: around 5-6 minutes (and this is after copying the table to the in_memory workspace). I'd really like to find a way to speed up the conversion to dictionary, or get some insight on a better strategy for manipulating large amounts of attribute data using Python.

UpdateCursors won't work well for me, because when one row changes, it has the potential to trigger changes in several others. Looping through and processing them one at a time is too cumbersome for what I need.

Answer 1

I think the problem is likely your two lines where you are going over the fields and appending each field individually to your subdict dictionary.

for field in valid_fields:
    subdict[field] = row[cursor_fields.index(field)]

Your row object is already a tuple in the same order as your fields, take advantage of that and use the zip function.

def make_attribute_dict(fc, key_field, attr_list=['*']):
    attdict = {}
    fc_field_objects = arcpy.ListFields(fc)
    fc_fields = [field.name for field in fc_field_objects if field.type != 'Geometry']
    if attr_list == ['*']:
        valid_fields = fc_fields
    else:
        valid_fields = [field for field in attr_list if field in fc_fields]
    #Ensure that key_field is always the first field in the field list
    cursor_fields = [key_field] + list(set(valid_fields) - set([key_field]))
    with arcpy.da.SearchCursor(fc, cursor_fields) as cursor:
        for row in cursor:
            attdict[row[0]] = dict(zip(cursor.fields,row))
    return attdict

This chugged through a 218k record 16 field file geodatabase feature class in 8 seconds on my system.

Edit: Tried a more rigorous test. 518k records over a remote SDE connection with 16 fields including OBJECTID and Shape, run at 32-bit. 11 seconds :)