Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

First, what am I trying to do?

I am building a Python script that should be called from an ArcMap 10.0 toolbox. This scripts uses (among other inputs) data contained in huge gdb tables. It processes the data in many different ways (hence, there are many inevitable iterations on the table).

Why would I want to convert the data?

The reason why I am converting the tables to ditionaries is that it is extremely slow to iterate through the table using SearchCursor. I have many operations that I want to do on my data and it is much simpler and faster to use existing data structures.

How am I doing it?

The usual way to use gdb tables in Python (and the one to which I have compared my method) is something along the lines of:

# Create a search cursor
rows = arcpy.SearchCursor(table)
for row in rows:
    (...)

What I'm doing instead, is that I am converting the table to dbf using TableToTable_conversion (throwing away some useless columns/rows on the way). Then I convert this DBF table to a dictionary (I was inspired by some code written by Tyrtamos, if you google "Lecture d'un fichier dbase III" you should find it). In the end, I have a dictionary containing lists of data indexed with the column names. And I'm very happy because I can do many things with it relatively quickly.

Finally,...

This method seems to be much faster. However, I am new to that game and I'm afraid to be missing something because I don't understand everything that's going on. Is it a bad idea to do that? Are there reasons not to do it other than potential memory overflow?

Please tell me if something is unclear. Thanks!

EDIT

After seeing your answers, I realize that there was one very important information missing: I'm using Arcmap 10.0.

I have run performance tests using nmpeterson's solution, modifying a few lines to make the code compatible with ArcGIS 10.0

cursor = arcpy.SearchCursor(fc, cursor_fields)
for row in cursor:
    rowList = []
    for field in cursor_fields:
        rowList.append(row.getValue(field))
    attr_dict[rowList[0]] = dict(zip(cursor_fields, rowList))

On a small table containing ~15000 rows, I measured an average elapsed time and got

make_attribute_dict -- 10.7378981873
gdb_to_dbf_to_dict-- 2.56576526461
share|improve this question
4  
I, for one, convert tables to dictionaries all the time in cases where frequent, somewhat-random access is required. –  nmpeterson May 16 at 17:02
    
If you could use arcpy.da cursors (available at ArcGIS 10.1 and up) I would say this particular procedure (not the concept in general) is a bad idea as it's overly complex and error-prone (GDB to DBF conversion is potentially lossy due to the the constraints of the DBF format). But if it works for you at the version you are using then more power to you. –  blah238 May 19 at 7:47
    
Then it's probably justified that I convert tables to dictionaries. @blah238 I am using ArcGIS 10.0, hence I don't have the data access module. Could you elaborate on the potential loss of the DBF format? What I've done seems to be working quite well, but I could be missing things –  Thibault May 19 at 7:59
    
Check my comment on @John's answer. –  blah238 May 19 at 8:02

2 Answers 2

Here's a function I've written to do a conversion directly from any feature class (or table) to a dictionary with an arcpy.da.SearchCursor:

def make_attribute_dict(fc, key_field, attr_list=['*']):
    ''' Create a dictionary of feature class/table attributes.
        Default of ['*'] for attr_list (instead of actual attribute names)
        will create a dictionary of all attributes. '''
    attr_dict = {}
    fc_field_objects = arcpy.ListFields(fc)
    fc_fields = [field.name for field in fc_field_objects if field.type != 'Geometry']
    if attr_list == ['*']:
        valid_fields = fc_fields
    else:
        valid_fields = [field for field in attr_list if field in fc_fields]
    # Ensure that key_field is always the first field in the field list
    cursor_fields = [key_field] + list(set(valid_fields) - set([key_field]))
    with arcpy.da.SearchCursor(fc, cursor_fields) as cursor:
        for row in cursor:
            attr_dict[row[0]] = dict(zip(cursor.fields, row))
    return attr_dict

In the resultant dictionary (call it attr_dict), the value of a specific field for a specific row (e.g. 'OBJECTID' = 5 when 'OBJECTID' was specified as the key_field) can be accessed with attr_dict[5]['FIELD_NAME'].

share|improve this answer
    
Your solution seems to be more elegant than mine. However I forgot to mention that I am working with ArcGIS 10.0, hence I don't have the data access module (arcpy.da). I have used your code to make it compatible with the version 10.0, but got much slower performances than with my previous gdb to DBF to dictionary solution. I will update my first post for more information. –  Thibault May 19 at 7:44

Probably not a complete answer as I'm not going to claim to be an absolute expert on the matter, but some things to think about:

I don't know what version of ArcGIS you are using, so I don't know if this would be applicable or not (not available in some of the older versions), but, if available, try the arcpy.da.searchcursor (http://resources.arcgis.com/en/help/main/10.1/index.html#//018w00000011000000) as my experience has told me there are SIGNIFICANT performance differences between the old .searchcursor and the new .da.searchcursor

When converting a GDB table to DBF table, unless the database design was done very carefully with this in mind, you are likely to have field names that get truncated due to DBF having max field name lengths. Also, you are likely to end up with some potentially unexpected results, if you're not careful, if you have sub-types and/or domains in use on your source data as, depending on your settings, it may export the coded value, the description value, or both.

share|improve this answer
5  
If you're using ArcGIS 10.1+, I would skip table conversion altogether and go straight from the original table to a dictionary using arcpy.da.SearchCursor(), from which you can directly specify a list of fields to keep, and optionally subset the records with a SQL query. –  nmpeterson May 16 at 17:00
    
I am using ArcGIS 10.0 and don't have this module...Concerning the GDB to DBF conversion, I am not quite sure of what you mean. For example, a field name "LONGFOO" could be truncated to "LONGF" ? And about the wrong export, could you give me an example where I could reproduce that? That sounds like a problem... –  Thibault May 19 at 7:50
    
DBF field names are limited to 10 characters and field lengths are limited to 255 characters: clicketyclick.dk/databases/xbase/format/dbf.html –  blah238 May 19 at 7:58
    
@blah238 Thanks, if that is it then that's not a problem. But I couldn't find information about the potentially unexpected results mentioned by John. –  Thibault May 19 at 8:14
    
I believe this is what John was referring to: forums.arcgis.com/threads/… Note that the environment setting to control this behavior was added at 10.1, so because you're at 10.0, there could be unexpected behavior that you can read about there (I am not familiar with it myself). –  blah238 May 19 at 8:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.