Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I was asking a question about Merging geodatabase previously, (link: Merge from arcpy.ListTables() producing duplicate rows?) and although the script worked, it took more than 9 hours to run when merging >3000 tables with 1 rows in each table. I was wondering whether instead of merging the tables it would be faster to write the contents of each geodatabase table into a csv file and keep appending tables to it. If so, how to do it?? I now need to merge tables of different geodatabase (it has 39 fields and varying number of rows) and I have a feeling that if I use the script from my previous question it will take even longer than 9 hours.

Basically, how to iterate through 3000 tables and write the attribute tables of each to the same csv and keep appending to it?

Here's the screenshot of the geodatabase tables I talked about above: enter image description here

share|improve this question

1 Answer 1

up vote 3 down vote accepted

With cursors and the csv module, this should go pretty quick:

import arcpy, csv, time

arcpy.env.workspace = <path to gdb>
table_list = arcpy.ListTables()

csv_out = <path to csv>

#Get name of fields from first entry
fields = [x.name for x in arcpy.ListFields(table_list[0])]   

start = time.time()
counter = 0

with open(csv_out, "wb") as f:
    wr = csv.writer(f)
    wr.writerow(fields)
    for table in table_list:
        with arcpy.da.SearchCursor(table, fields) as cursor:
            for row in cursor:
                counter += 1
                wr.writerow(row)

end = time.time()- start

print "{0} rows from {1} tables written in {2} seconds".format(counter, len(table_list), end)

On my machine, this was the output from tables with 26 fields:

3636 rows from 101 tables written in 7.33599996567 seconds.

share|improve this answer
    
thanks so much! I no longer have access to computer arcgis and these data, but I'll try your suggestion tomorrow. would that be okay with you if I ask you again IFF something is not going to work as expected? –  atoregozh Jul 10 at 22:01
    
@atoregozh, sure! I updated the script a bit to show the run time. Should eat through 3000 tables in no time. –  Paul Jul 10 at 22:03
    
thanks again! how do get 101 pseudo tables with 26 fields? –  atoregozh Jul 10 at 22:08
    
I just scripted a solution to copy the same table 100 times, haha. I still iterated over every table though. –  Paul Jul 10 at 22:10
2  
Using "with...as" disposes of the object after you are done with it. In ArcGIS, not disposing stuff sometimes creates issues dealing with memory management. Using "with...as" provides a nice clean way to use a resource and automatically get rid/release it when you are done with it. It's a cleaner method than a try/finally block: effbot.org/zone/python-with-statement.htm –  Conor Jul 10 at 22:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.