The following script just copies a file into RAM, performs some simple operations and writes the result back to hard disc:
import arcpy import random import time arcpy.management.Delete("in_memory") input_table = "C:\\data\\entrypoints\\table.shp" memory_table = "in_memory\\table1" # input from disc t0 = time.clock() arcpy.management.CopyRows(input_table, memory_table) print time.clock() - t0 # some examplary operations and conversions arcpy.management.AddField(memory_table, "NewField", "FLOAT") tab = arcpy.da.TableToNumPyArray(memory_table, "*") for row in tab: row = random.random() # output to disc output_table = "output_table" t0 = time.clock() arcpy.da.NumPyArrayToTable(tab, output_table) print time.clock() - t0 arcpy.management.Delete("in_memory")
I used this script on a table with 150K rows and 4 integer columns (~5MB filesize) stored on a SSD. Reading a binary file of this size takes less than 1 second in C++ or Python. However, in ArcPy
CopyRows(...) takes around 5 minutes (measured with Python's
time.clock()). For the output, the performance is similarly bad. I am quite puzzled about this.
I already tried other alternatives to the input (e.g. using UpdateCursor and da.UpdateCursor but I could not achieve similar times). Furthermore, I read about running the script externally, but this makes things worse. The script above is as fast as I can get.
Am I doing something terribly wrong here? How come file-IO is so slow in ArcPy? Is there any solution to this or do I have to write a convenient IO by myself?
Edit: As hinted by @Jason Scheirer, the result of the internal computations can be written directly to disc, no need to go via internal memory again.
Update: I had the problem checked on a alternative environment by a client (organization with >50 Arc* users). In their native Windows environment the performance was equally bad. So I assumed the problem was not related to my system. However, it turned out that on my Windows inside a virtual machine the problem is solved by keeping the data on a local hard drive, dedicated to the vm. I never experienced performance issues with other applications accessing the parent (or a remote) file system. So I was misguided, sorry for that! I am curious what causes the bad performance on the client organization's environment... Will adapt the title and description ASAP.