1

I have been attempting to append 10,000 tables using the following arcpy code (from an answer to Appending 10,000 tables using ArcPy?):

WD = "C:/"  
arcpy.env.workspace = WD

for table in arcpy.ListTables():
    if not table.startswith('test_4'):
        tableData = list(arcpy.da.SearchCursor(table, "*"))
        cursor = arcpy.da.InsertCursor("test_4", "*")
        for data in tableData:
        cursor.insertRow(data)

This code works perfectly for 5 or 500 tables. When I try to append 10,000 tables, the program runs for 5 minutes and then stops running. It does not give an error message, however. The program seems to still be running, although it is in fact not.

What could be causing this?

  • 1
    is this a standalone script? or within the pywin in arc? – ziggy Sep 28 '16 at 16:26
  • 4
    No idea. Are you using 64 bit python? It is recommended you use the "with" syntax for cursors so that python deletes and does garbage collection for each iteration of cursor. Finally, try chunking your table list and using subprocess in your iteration. That will spawn a new process for each chunk of append statements. When the append finishes, the process will release the memory and hopefully keep it from crashing. It might be a bit slower but may actually complete. – Ben S Nadler Sep 28 '16 at 16:27
  • 2
    The question is tagged ArcGIS-10.0, but your code features DA cursors, which weren't available until 10.1. What version of AcGIS are you really using? – Vince Sep 28 '16 at 17:27
  • 1
    Why convert the SearchCursor to a list? Try something like: for row in arcpy.da.SearchCursor(table, "*")): cursor.insertRow(row) Converting it to a list first is going to require the whole thing to be saved in memory. I believe the da cursors have something in place to allow you to iterate over larger features. – eseglem Sep 28 '16 at 20:22
  • 2
    @gis_user Please edit your question to make it clearer rather than adding numerous comments. It is difficult to follow through the comments along with your question to understand the problem and what you've tried. – Midavalo Sep 28 '16 at 22:46

Browse other questions tagged or ask your own question.