Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

A while ago, I wrote a quick Python function for converting an attribute table to a python dictionary, where the key is taken from a user-specified unique ID field (typically the OID field). Additionally, by default all fields are copied to the dictionary, but I've included a parameter allowing for just a subset to be specified.

def make_attribute_dict(fc, key_field, attr_list=['*']):
    dict = {}
    fc_field_objects = arcpy.ListFields(fc)
    fc_fields = [field.name for field in fc_field_objects if field.type != 'Geometry']
    if attr_list == ['*']:
        valid_fields = fc_fields
    else:
        valid_fields = [field for field in attr_list if field in fc_fields]
    if key_field not in valid_fields:
        cursor_fields = valid_fields + [key_field]
    else:
        cursor_fields = valid_fields
    with arcpy.da.SearchCursor(fc, cursor_fields) as cursor:
        for row in cursor:
            key = row[cursor_fields.index(key_field)]
            subdict = {}
            for field in valid_fields:
                subdict[field] = row[cursor_fields.index(field)]
            dict[key] = subdict
            del subdict
    return dict

This works great for relatively small datasets, but I just ran it on a table containing about 750,000 rows and 15 fields -- around 100MB in a file geodatabase. On these, the function runs much slower than I would have expected: around 5-6 minutes (and this is after copying the table to the in_memory workspace). I'd really like to find a way to speed up the conversion to dictionary, or get some insight on a better strategy for manipulating large amounts of attribute data using Python.

UpdateCursors won't work well for me, because when one row changes, it has the potential to trigger changes in several others. Looping through and processing them one at a time is too cumbersome for what I need.

share|improve this question
2  
The limiting factor in how much you can optimize your script might be the length of time it takes to iterate through your cursor. Have you compared the time it takes to iterate through the cursor without building your dictionaries? –  Jason Mar 18 '13 at 21:08
2  
@Jason commenting out the lines from subdict = {} through del subdict yields a processing time of about 10 seconds. –  nmpeterson Mar 18 '13 at 21:18
    
You probably know more about this than I, but the only other thing I would offer in terms of optimization is looking at whether calling subdict[field] = row[cursor_fields.index(field)] is faster than calling subdict[field] = row.getValue(field). In the latter scenario you would be performing one step...though the difference in performance between indexing two lists (cursor_fields and row) and using a single ESRI process may not be much better and might just even be worse! –  Jason Mar 18 '13 at 21:30
add comment

1 Answer

up vote 12 down vote accepted

I think the problem is likely your two lines where you are going over the fields and appending each field individually to your subdict dictionary.

for field in valid_fields:
    subdict[field] = row[cursor_fields.index(field)]

Your row object is already a tuple in the same order as your fields, take advantage of that and use the zip function.

def make_attribute_dict(fc, key_field, attr_list=['*']):
    attdict = {}
    fc_field_objects = arcpy.ListFields(fc)
    fc_fields = [field.name for field in fc_field_objects if field.type != 'Geometry']
    if attr_list == ['*']:
        valid_fields = fc_fields
    else:
        valid_fields = [field for field in attr_list if field in fc_fields]
    #Ensure that key_field is always the first field in the field list
    cursor_fields = [key_field] + list(set(valid_fields) - set([key_field]))
    with arcpy.da.SearchCursor(fc, cursor_fields) as cursor:
        for row in cursor:
            attdict[row[0]] = dict(zip(cursor.fields,row))
    return attdict

This chugged through a 218k record 16 field file geodatabase feature class in 8 seconds on my system.

Edit: Tried a more rigorous test. 518k records over a remote SDE connection with 16 fields including OBJECTID and Shape, run at 32-bit. 11 seconds :)

share|improve this answer
1  
Note that I made key_field the first field so that I could rely on using row[0] to reference the value of key_field. I also had to change your variable dict to attdict. dict is a keyword, and without that keyword I could not use dict(zip()) –  blord-castillo Mar 18 '13 at 21:42
5  
Clever. This is exactly the kind of sweet idiomatic Python that arcpy.da is meant to enable. –  Jason Scheirer Mar 18 '13 at 23:09
    
Great insight. Love the method, and it really helped. –  nmpeterson Mar 19 '13 at 14:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.