Importing ArcPy taking many seconds and code is really slow for GeoProcessing Tools 64-bit (ArcGIS 10.1 SP1)

Question

I've installed ArcGIS10.1 SP1 and whenever I import arcpy (on Pywin console or Eclipse PyDev) it takes aprox. 15-20seconds. Is this normal? I don't think so...

I had Python 2.7.3 64-bit previously installed on my Windows 7 (and still use it) and now I also have Python 32-bit and Python 64-bit that came with ArcGIS. I think I might have the paths confused, because the 64bit version of Python 2.7 in the folder C:\Python27\ArcGISx6410.1was supposed to be 2.7.2, but when I check the version, inside that path, it gives me:

C:\Python27\ArcGISx6410.1>python.exe Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win 32

My %PATH% has this, among other paths:

...
C:\Python27
C:\Python27\Scripts
C:\Program Files (x86)\ArcGIS\Desktop10.1\Bin
C:\Program Files (x86)\ArcGIS\EsriProductionMapping\Desktop10.1\Bin
...

And I didn't need to set PYTHONPATH. My ArcMap Python shell works fine and so does the 64-bit version for the backgorund scripting.

My code is very simple: basically I have to find how many times does a node (NO) from a node feature class appears in other line feature class. It works, but for #6290 nodes, it takes more than 10min!

'''

# Import system modules
import arcpy, sys
from arcpy import env

print "Running against: {}".format(sys.version)

def do_search_by_cursor(table,searchString):
    count = 0
    count += cursor_search_count(table,searchString)    
    #print "done by cursor: " + str(count)
    return count

def cursor_search_count(table,searchstring):
    rows = arcpy.SearchCursor(table,searchstring)
    count = 0
    for row in rows:
        count += 1
    return count

# Set the workspace environment to local file geodatabase
env.workspace = "S:/workspace/MSc_git/ArcGis/estagio_db.gdb"

# Set local variables
fc_nodes = "Rede_Cbr_MM_node_proj"
fc_links = "Rede_Cbr_MM_2008_WithoutMetro"
# fieldName1 = "LEGS"
# fieldAlias = "numLegs"

fieldname="TONODENO"
# Create field name with the proper delimiters
delimitedfield = arcpy.AddFieldDelimiters(fc_links, fieldname)

# Create search cursor:
rows = arcpy.SearchCursor(fc_nodes, "", "", fields="NO")

for row in rows:
    searchStr= delimitedfield +" = " +str(row.NO)
    myvar= do_search_by_cursor(fc_links, searchStr)
    print "Node %d has %d legs" %(row.NO,myvar)

My PyDev is configured to use the specific Python Interpreter that came with ArcGIS processing Tools 64-bit and looks like this (I've checked all...):

If you could give a hint or a link (I've already explored, but nothing seems to work), I'll appreciate it.

Answer 1

I also create ArcPy geoprocessing tools using PyDev for Eclipse. Generally, I run my scripts as in-process geoprocessing tools in a custom toolbox (.tbx) from within ArcMap so that the import arcpy happens instantly. If I run a script from outside of ArcMap, the import arcpy step usually takes about 5 seconds on a Sandy Bridge Core i5.

I think your trouble may lie in the use of 64-bit Python and the ArcGIS 64-bit background geoprocessing package. I don't know why this would take so much longer to import compared to 32-bit ArcPy. If you want confirmation, you could iteratively add/remove Python libraries from the Python Interpreters menu you have and see if a specific folder is causing the excessive import times.

Personally, I haven't found a compelling need to use 64-bit geoprocessing. Even in some of the massive Spatial Analyst workflows I've had to implement, geoprocessing usually crashes before I hit memory limits. I prefer to divide my inputs before processing and merge them together at the end. This also ensures better compatibility with 3rd part modules, most of which are already compiled for 32-bit Python.

---

I also think there's a bunch of tweaks you can make to your code to make it faster, besides the import arcpy issue and 64- vs. 32-bit Python. Creating a new searchCursor object for each iteration of your fc_nodes features is really slow. This means you are creating 6290 * 6290 = 39,564,100 search cursor objects. Creating each cursor object has processing overhead, so it's not surprising that this is taking many minutes to run. You want to avoid this code structure with nested cursors.

It appears that you are trying to find the counts of features in fc_links with the value of the TONODENO field equal to each possible value of the NO field in fc_nodes. Instead of using these nested cursors, it would be much easier to use the arcpy.Statistics_analysis() tool, with the TONODENO as the case field:

fc_links = "Rede_Cbr_MM_2008_WithoutMetro"
fieldname = "TONODENO"
statisticsTable = "OutputStatisticsTable"
arcpy.Statistics_analysis(fc_links, statisticsTable, [[fieldname,"COUNT"]], fieldname)

This is similar to do a Group By query in SQL, or a Totals query in Excel. In the output table, the FREQUENCY column gives the number of features in fc_links that have that value for the TONODENO field. You can join this back table to the fc_nodes table, or store it in a dictionary or something like that.

Answer 2

The import requests an ArcGIS license and it can be the licence manager causing the delay. Especially for a standalone (non-toolbox) script.

I am seeing 7 seconds delay to import arcpy, whereas all the other imports are less than 1 second.

One suggestion is to 'borrow a license and then clear out the license manager name in ArcGIS Administrator.' See https://community.esri.com/thread/209391-arcpy-import-is-really-slow-2-minutes

Answer 3

I've updated my code using @dmahr sugestions and it's much faster, no doubts! I've tracked the import time for arcpy...it takes more than 23 seconds and the code only around 7s!

I think it's interesting:

import time
import sys
t_start = time.clock()
import arcpy
from arcpy import env, da
arcpy.AddMessage("Importing arcpy outside ArcMap took %s seconds." % (time.clock() - t_start))
from cProfile import run


def main():
    print "Running against: {}".format(sys.version)
    t_start = time.clock()
    # Set the workspace environment to local file geodatabase
    env.workspace = "S:/workspace/MSc_git/ArcGis/estagio_db.gdb"

    # Set local variables for feature classes, tables or shapefiles
    fc_nodes = "Rede_Cbr_MM_node_proj"
    fc_links = "Rede_Cbr_MM_2008_WithoutMetro"
    # fieldName1 = "LEGS"
    # fieldAlias = "numLegs"
    #
    # # Execute AddField twice for two new fields
    # arcpy.AddField_management(fc_nodes, fieldName1, "SHORT", "", "", "",
    #                           fieldAlias, "NULLABLE")

    try:
        fieldname = "TONODENO"
        statisticsTable = "Stats_Links_count_TONODENO"
        arcpy.Statistics_analysis(fc_links, statisticsTable, [[fieldname, "COUNT"]], fieldname)
        # arcpy.JoinField_management(fc_nodes,"NO",statisticsTable,"TONODENO","COUNT TONODENO")

        out_dict = {}
        with da.SearchCursor(statisticsTable, ['TONODENO', 'COUNT_TONODENO']) as cursor:
            for row in cursor:
                #print "No %s has %s legs" %(row[0],row[1])
                out_dict[row[0]]= row[1]
        #print out_dict

    except Exception, e:
        # If an error occurred, print line number and error message
        tb = sys.exc_info()[2]
        print "Line %i" % tb.tb_lineno
        print e.message

    arcpy.AddMessage("      ... complete in %s seconds." % (time.clock() - t_start))

if __name__ == '__main__':
    run("main()")

And the result is:

Importing arcpy outside ArcMap took **23.546973162 seconds.**
Running against: 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)]
(...)
 ... complete in 7.91369121857 seconds.
     26 function calls in 7.915 seconds

Than I've created a dictionary because I'll need to check some conditions about the node number and update the other Feature Class (fc_links) accordingly, like this:

import time
import sys
t_start = time.clock()
import arcpy
from arcpy import env, da
arcpy.AddMessage("Importing arcpy outside ArcMap took %s seconds." % (time.clock() - t_start))
from cProfile import run


def main():
    print "Running against: {}".format(sys.version)
    t_start = time.clock()
    # Set the workspace environment to local file geodatabase
    env.workspace = "S:/workspace/MSc_git/ArcGis/estagio_db.gdb"

    # Set local variables for feature classes, tables or shapefiles
    fc_nodes = "Rede_Cbr_MM_node_proj"
    fc_links = "Rede_Cbr_MM_2008_WithoutMetro"
    # fieldName1 = "LEGS"
    # fieldAlias = "numLegs"
    #
    # # Execute AddField twice for two new fields
    # arcpy.AddField_management(fc_nodes, fieldName1, "SHORT", "", "", "",
    #                           fieldAlias, "NULLABLE")

    try:
        fieldname = "TONODENO"
        statisticsTable = "Stats_Links_count_TONODENO"
        # arcpy.Statistics_analysis(fc_links, statisticsTable, [[fieldname, "COUNT"]], fieldname)
        # arcpy.JoinField_management(fc_nodes,"NO",statisticsTable,"TONODENO","COUNT TONODENO")

        out_dict = {}
        # Create search update_cursor for the statistic table previously created
        with da.SearchCursor(statisticsTable, ['TONODENO', 'COUNT_TONODENO']) as cursor:
            for row in cursor:
                #print "No %d = %d" %(row[0],row[1])
                #Create a dictionary for counting how many times each TONODENO appeared in fc_links
                #nodeNr "TONODENO" appears n times
                out_dict[int(row[0])] = int(row[1])
            del row
        del cursor

        # Create update update_cursor for feature class of nodes 
        with da.UpdateCursor(fc_nodes, ['NO', 'LEGS']) as update_cursor:
            for record in update_cursor: 
                node = record[0]       
                #If no is not on dic, it means it a POI, like a bus stop, not an intersection               
                if node not in out_dict.keys():
                    print "--NOT IN DIC: Node nr %d " %(node)
                    record[1] = -1
                else:              
                    if out_dict.get(node) == 3:
                        # Node represents a T-Intersection
                        record[1] = 0
                        print "#Node nr %d = %d " %(node,record[1])
                    elif out_dict.get(node)  == 4:
                        # # Node represents a Cross-Intersection                       
                        record[1] = 1
                        print "*Node nr %d = %d" %(node,record[1])
                    else:
                        # Node represents another thing (e.g. bus stop,...)
                        print "Node nr %d NOT an intersection with >=3 legs" %(node)
                        record[1] = -1                     
                # Update the update_cursor with the updated list       
                update_cursor.updateRow(record)
            del record
        del update_cursor 

    except Exception, e:
        # If an error occurred, print line number and error message
        tb = sys.exc_info()[2]
        print "Line %i" % tb.tb_lineno
        print e.message

    arcpy.AddMessage("      ... complete in %s seconds." % (time.clock() - t_start))

if __name__ == '__main__':
    run("main()")

And it took less time importing arcpy for the 2nd time (I really don't get this big difference...):

Importing arcpy outside ArcMap took 8.96872025649 seconds.
Running against: 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)]