IO Performance of ArcPy in virtual machine

Question

The following script just copies a file into RAM, performs some simple operations and writes the result back to hard disc:

import arcpy
import random
import time

arcpy.management.Delete("in_memory")
input_table = "C:\\data\\entrypoints\\table.shp"
memory_table = "in_memory\\table1"

# input from disc
t0 = time.clock()
arcpy.management.CopyRows(input_table, memory_table)
print time.clock() - t0

# some examplary operations and conversions
arcpy.management.AddField(memory_table, "NewField", "FLOAT")
tab = arcpy.da.TableToNumPyArray(memory_table, "*")
for row in tab:
  row[0] = random.random()

# output to disc
output_table = "output_table"
t0 = time.clock()
arcpy.da.NumPyArrayToTable(tab, output_table)
print time.clock() - t0

arcpy.management.Delete("in_memory")

I used this script on a table with 150K rows and 4 integer columns (~5MB filesize) stored on a SSD. Reading a binary file of this size takes less than 1 second in C++ or Python. However, in ArcPy CopyRows(...) takes around 5 minutes (measured with Python's time.clock()). For the output, the performance is similarly bad. I am quite puzzled about this.

I already tried other alternatives to the input (e.g. using UpdateCursor and da.UpdateCursor but I could not achieve similar times). Furthermore, I read about running the script externally, but this makes things worse. The script above is as fast as I can get.

Am I doing something terribly wrong here? How come file-IO is so slow in ArcPy? Is there any solution to this or do I have to write a convenient IO by myself?

Edit: As hinted by @Jason Scheirer, the result of the internal computations can be written directly to disc, no need to go via internal memory again.

Update: I had the problem checked on a alternative environment by a client (organization with >50 Arc* users). In their native Windows environment the performance was equally bad. So I assumed the problem was not related to my system. However, it turned out that on my Windows inside a virtual machine the problem is solved by keeping the data on a local hard drive, dedicated to the vm. I never experienced performance issues with other applications accessing the parent (or a remote) file system. So I was misguided, sorry for that! I am curious what causes the bad performance on the client organization's environment... Will adapt the title and description ASAP.

Answer 1

I just created a shapefile of 150k features, 4-column integers and I was able to copy it to memory in 6 seconds using this:

import os
import arcpy
from random import randint
import time

newshape = "C:/users/paul/desktop/test.shp"
fields = ["w", "x", "y", "z"]
rows = 150000
path,name = os.path.split(newshape)

arcpy.CreateFeatureclass_management(path, name, "POINT")
[arcpy.AddField_management(newshape, field, "LONG") for field in fields]

with arcpy.da.InsertCursor(newshape, fields) as Icursor:
    for _ in xrange(rows):
        Icursor.insertRow([randint(0,rows), randint(0,rows),
                           randint(0,rows), randint(0,rows)])

timer = time.time()
arcpy.management.CopyRows(newshape, "in_memory/test")
print time.time() - timer

I'm not using an SSD, but I'm on a pretty powerful workstation. (6 core 2.4GHz Xeon, 12GB RAM).

Not that this seems to be slowing you down as of yet, but you'll get faster times if you utilize from xxx import yyy:

from random import random
tab = arcpy.da.TableToNumPyArray(memory_table, "*")
for row in tab:
  row[0] = random()

timeit is great for timing two pieces of code:

python -m timeit -s "import random" "random.random()"
10000000 loops, best of 3: 0.138 usec per loop

python -m timeit -s "from random import random" "random()"
10000000 loops, best of 3: 0.082 usec per loop

python -m timeit -s "from random import random as rand" "rand()"
10000000 loops, best of 3: 0.0836 usec per loop

Yes, we're discussing a barely noticeable difference over 150k iterations (on my laptop it's about .01 sec). Yes, it's premature optimization but if I'm going to be calling a function inside a loop, it's a very simple change. You can even use from xxx import yyy as zzz so there isn't a clash in the namespace.