3

The following script just copies a file into RAM, performs some simple operations and writes the result back to hard disc:

import arcpy
import random
import time

arcpy.management.Delete("in_memory")
input_table = "C:\\data\\entrypoints\\table.shp"
memory_table = "in_memory\\table1"

# input from disc
t0 = time.clock()
arcpy.management.CopyRows(input_table, memory_table)
print time.clock() - t0

# some examplary operations and conversions
arcpy.management.AddField(memory_table, "NewField", "FLOAT")
tab = arcpy.da.TableToNumPyArray(memory_table, "*")
for row in tab:
  row[0] = random.random()

# output to disc
output_table = "output_table"
t0 = time.clock()
arcpy.da.NumPyArrayToTable(tab, output_table)
print time.clock() - t0

arcpy.management.Delete("in_memory")

I used this script on a table with 150K rows and 4 integer columns (~5MB filesize) stored on a SSD. Reading a binary file of this size takes less than 1 second in C++ or Python. However, in ArcPy CopyRows(...) takes around 5 minutes (measured with Python's time.clock()). For the output, the performance is similarly bad. I am quite puzzled about this.

I already tried other alternatives to the input (e.g. using UpdateCursor and da.UpdateCursor but I could not achieve similar times). Furthermore, I read about running the script externally, but this makes things worse. The script above is as fast as I can get.

Am I doing something terribly wrong here? How come file-IO is so slow in ArcPy? Is there any solution to this or do I have to write a convenient IO by myself?

Edit: As hinted by @Jason Scheirer, the result of the internal computations can be written directly to disc, no need to go via internal memory again.

Update: I had the problem checked on a alternative environment by a client (organization with >50 Arc* users). In their native Windows environment the performance was equally bad. So I assumed the problem was not related to my system. However, it turned out that on my Windows inside a virtual machine the problem is solved by keeping the data on a local hard drive, dedicated to the vm. I never experienced performance issues with other applications accessing the parent (or a remote) file system. So I was misguided, sorry for that! I am curious what causes the bad performance on the client organization's environment... Will adapt the title and description ASAP.

  • 1
    This sounds like a fluke. Is it consistently slow like this? Across multiple machines? Have you tried the Copy tool or the Append tool? Do they benchmark similarly? – Jason Scheirer Oct 11 '13 at 15:59
  • I suspected system dependencies too, and tested this program on two different machines (Win7 inside VirtualBox and native WinXP System), but got similar times. Regarding Copy and Append: From the documentation "The 'in_memory' workspace is not supported as the output location." It raises Error 260. – jotrocken Oct 14 '13 at 11:24
  • Okay, so the problem is the execution in this one specific tool is slow in this one specific environment, not that all I/O in arcpy tools is slow. Is there really a reason you're temporarily shuttling from numpy→in memory→disk? Couldn't you go from numpy array to disk directly or are you doing additional work in the interim before finally flushing out? – Jason Scheirer Oct 14 '13 at 16:40
  • You are right: The "result" of the numpy operations can be written to disc directly. However, this does not change the speed of read and write operations. For explanation, this script is intended to be an entry point for a series of computations written in Python, so I need the data in pythonic structures. Which will probably be numpy arrays for space reasons. – jotrocken Oct 14 '13 at 17:10
  • What type of spatial objects are you using? What projection? I/O times between PolygonZ and Point shapefiles can be very different, and might account for the differences between your experience and @Jason-Scheirer. I suspect projection could be an issue as well, but not sure on that. Also, have you built shape indexes for your shapefile? Minimum bounding rectangle (and associated precision) could account for differences. – blord-castillo Oct 14 '13 at 17:30
4

I just created a shapefile of 150k features, 4-column integers and I was able to copy it to memory in 6 seconds using this:

import os
import arcpy
from random import randint
import time

newshape = "C:/users/paul/desktop/test.shp"
fields = ["w", "x", "y", "z"]
rows = 150000
path,name = os.path.split(newshape)

arcpy.CreateFeatureclass_management(path, name, "POINT")
[arcpy.AddField_management(newshape, field, "LONG") for field in fields]

with arcpy.da.InsertCursor(newshape, fields) as Icursor:
    for _ in xrange(rows):
        Icursor.insertRow([randint(0,rows), randint(0,rows),
                           randint(0,rows), randint(0,rows)])

timer = time.time()
arcpy.management.CopyRows(newshape, "in_memory/test")
print time.time() - timer

I'm not using an SSD, but I'm on a pretty powerful workstation. (6 core 2.4GHz Xeon, 12GB RAM).

Not that this seems to be slowing you down as of yet, but you'll get faster times if you utilize from xxx import yyy:

from random import random
tab = arcpy.da.TableToNumPyArray(memory_table, "*")
for row in tab:
  row[0] = random()

timeit is great for timing two pieces of code:

python -m timeit -s "import random" "random.random()"
10000000 loops, best of 3: 0.138 usec per loop

python -m timeit -s "from random import random" "random()"
10000000 loops, best of 3: 0.082 usec per loop

python -m timeit -s "from random import random as rand" "rand()"
10000000 loops, best of 3: 0.0836 usec per loop

Yes, we're discussing a barely noticeable difference over 150k iterations (on my laptop it's about .01 sec). Yes, it's premature optimization but if I'm going to be calling a function inside a loop, it's a very simple change. You can even use from xxx import yyy as zzz so there isn't a clash in the namespace.

  • "you'll get faster times if you utilize from xxx import yyy" -- proof? – blah238 Oct 12 '13 at 6:14
  • @blah238, updated my code. Yes, I realize the claim is a bit silly since "faster" at such small times really is unimportant, but I figured I'd toss it in to make my post longer and a little more informative, haha. – Paul Oct 12 '13 at 16:07
  • I meant proof that the function call itself would be faster, not the import statement (which you would never call in a loop in real code). – blah238 Oct 12 '13 at 17:03
  • Never mind, I see that the import is being called just once as the setup argument. – blah238 Oct 12 '13 at 17:06
  • Yes, I know about timeit and I would definitely not use it for code which runs longer than 1 minute. The speed of the numpy code is not the matter here (runs in about 2.5s), nor will the random()-call be in the final code. It's there to demonstrate that there is some operation going on between input and output. I left the time.clock() away from the minimal working example, however I re-add it to point out that the issue is within IO. – jotrocken Oct 14 '13 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.