5

Is there something happening on the back-end, that makes the process below so slow, that can be avoided to speed it up?

It is taking between 5 and 7 seconds to turn one layer file (.lyr) into a layer object using layer = arcpy.mapping.Layer(path). This is painfully slow when I have over 2,500 layers that I need to convert to layer objects so that I can access their datasetName property.

Within a directory of layers, I want to be able to search for all layers that point to a given dataset in our SDE. For example, NYC's Planimetrics feature classes contain the word "NYCMAP," and I want to search for all layer files that are currently pointing to any feature classes containing that word.

My current script actually works fine, but it's painfully slow. So I decided to break it down line by line to try to find/understand the snag. It didn't take long for me to find it...

for dirname, dirnames, filenames in os.walk(layersFolder):

    for filename in filenames:
        path = os.path.join(dirname, filename)
        filepath, extension = os.path.splitext(path)

        if extension == ".lyr":
            # slow down happening here:
            layer = arcpy.mapping.Layer(path)
            print layer

If I run these few lines before converting to a layer object, I run through over 2,500 layers in about 15 seconds. When I include the layer = arcpy.mapping.Layer(path) line, it slows to about 3 1/2 hours.

I have tried using arcpy.walkinstead of os.walk, and this has no effect. I have also tried copying the layers directory and my script to my C:\ (as opposed to having both on our network), but this also has no effect.

Using Python 2.7.10 and ArcGIS 10.4


UPDATE:

I thought I would include my entire working code. So this code currently takes about 3 hours and 15 minutes to go through just under 3,000 layer files. Rather than search for a specific keyword each time, I simply output the entire directory of layers into an Excel doc, and then just search inside of that doc. I will eventually schedule the script to run about once a week or so. Someone suggested created a database table and using SQL to query, which is an interesting idea that I may explore.

import sys
sys.path.append(r'somePath')            # Looking in our Python Library for the xlsxwriter module
import arcpy, os, xlsxwriter, string
from time import strftime
from datetime import datetime

# Set time & date variables
startTime = datetime.now()              # Used at end of script to calculate the time it took the script to run
dateStr = strftime('%m/%d/%Y %H:%M')    # Variable for the current date/time (printed below)
dateNameStr = strftime('%Y%m%d')        # Used later in output file name
timeStr = strftime('%H%M')              # Used later in output file name
print "{}\n".format(dateStr)

outputFileFolder = r'somePath'

# ----------------------------------------------------------------------
# Functions
# ----------------------------------------------------------------------
def getColumnList(someList):
    '''Creates a list of Excel columns (Note: this function is restricted to 26 columns (i.e. A->Z))'''
    columnList = []
    n = 0
    alphaList = list(string.ascii_uppercase)
    someListLen = len(someList)
    while n <= someListLen:
        columnList.append(alphaList[n])
        n += 1
    return columnList

# ----------------------------------------------------------------------
# Walk through layers and get layer attributes
# ----------------------------------------------------------------------
# Layers folder
layersFolder = r'somePath'
print "Collecting attributes from all layers in {}\n".format(layersFolder)

layersList = []
walk = arcpy.da.Walk(layersFolder, datatype="Layer")
for dirpath, dirnames, filenames in walk:
    for filename in filenames:
        print filename                              
        path = os.path.join(dirpath, filename)
        layer = arcpy.mapping.Layer(path)

        layerList = []

        if layer.isGroupLayer:
            for sublayer in layer:
                layerList.append(sublayer)
        else: layerList.append(layer)

        for singleLayer in layerList:

            if singleLayer.supports("datasetName"):
                layerInfo = (os.path.split(path)[0], os.path.split(path)[1], str(singleLayer), singleLayer.datasetName)
                print str([str(item) for item in layerInfo])
                layersList.append(layerInfo)

    print

# ----------------------------------------------------------------------
# Create Excel doc using XLSXWriter
# ----------------------------------------------------------------------
print("Creating Excel doc...\n")

# Create Excel doc
workbook = xlsxwriter.Workbook(os.path.join(outputFileFolder, "Layers_and_Datasets_{}_{}.xlsx".format(dateNameStr, timeStr)))
worksheet = workbook.add_worksheet()

# Format the worksheet
worksheet.set_column('A:A', 50)                 # Set column width
worksheet.set_column('B:B', 50)                 # Set column width
worksheet.set_column('C:C', 50)                 # Set column width
worksheet.set_column('D:D', 50)                 # Set column width
bold = workbook.add_format({'bold': True})      # Create bold format object for column headers

# Column headers/column list
headers = ("Layer Path", "Layer Name", "Sublayer Heirarchy", "Dataset Name")
columnList = getColumnList(headers)

# Write headers in cells A1, B1, C1, & D1
i = 0
for header in headers:
    worksheet.write(columnList[i] + "1", str(headers[i]), bold)
    i += 1

# Sort the layersList list
sortedlayersList = sorted(list(layersList), key=lambda layerPath: layerPath[0])

# Write all other rows, beginning in row 2
row = 2
for layerRow in sortedlayersList:
    attr = 0                                    # "attr" stands for layer attribute
    for lyrAttr in layerRow:
        worksheet.write(str(columnList[attr])+str(row), lyrAttr)
        attr += 1
    row += 1

# Close workbook
workbook.close()

# ----------------------------------------------------------------------
# Finish script
# ----------------------------------------------------------------------
scriptTime = datetime.now() - startTime
print "{}\n".format(scriptTime)

print 'Script Complete'
  • I've been doing something similar this week. I noticed that reading layer files created in version 9 take longer than those created in v 10 – Bjorn Dec 7 '16 at 22:24
  • Hey @Bjorn. I should have mentioned that I am actually using Python 2.7.10 and ArcGIS 10.4, although I'm not sure it matters much. – Kristen G. Dec 7 '16 at 22:54
  • Out of curiosity have you tried env.workspace=r'in_memory' – FelixIP Dec 7 '16 at 22:57
  • Please edit the question to provide additional details -- it's not fair to those who would answer to need to mine the comments for critical information. The exact version of software should always be included in a question. Some layer files require a statistical scan of a source table to determine Natural Breaks. The performance of such a scan is linked to the feature count. – Vince Dec 8 '16 at 11:58
  • @Vince Edited to add versions directly to the question. Are you saying that layers pointing to datasets with higher feature counts will take longer to convert into layer objects? – Kristen G. Dec 8 '16 at 15:48
2

Short answer

Creating an arcpy.mapping.Layer object for a .lyr file that references an SDE feature class when running a Python script not on the same machine where you have your SDE geodatabase takes several seconds. The performance is poor because it takes a longer time to access the SDE geodatabase over the network.

Solution: run your Python script on the same machine where you have your SDE geodatabase. Then it takes fractions of a second for every .lyr file.

Long answer

I have done a simple benchmark to see if creating of Layer objects can be sped up. And it seems as you cannot do anything about it really.

The code to run:

## -*- coding: UTF-8 -*-
from __future__ import print_function

import os
import sys
print(sys.version)
import arcpy
import time
from functools import wraps

def report_time(func):
    '''Decorator reporting the execution time'''
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print(func.__name__, round(end-start,3))
        return result
    return wrapper

@report_time
#----------------------------------------------------------------------
def get_filepath(input_folder):
    for dirname, dirnames, filenames in os.walk(input_folder):
        for filename in filenames:
            path = os.path.join(dirname, filename)
            filepath, extension = os.path.splitext(path)
            if extension == ".lyr":
                # slow down happening here:
                print(filepath)

                layer = create_lyr(path)
                get_dataset_name(layer)

@report_time
#----------------------------------------------------------------------
def create_lyr(path):
    if 'Continuum' in sys.version:
        layer = arcpy.mp.LayerFile(path)
    else:
        layer = arcpy.mapping.Layer(path)
    return layer

@report_time
#----------------------------------------------------------------------
def get_dataset_name(layer):
    if 'Continuum' in sys.version:
        l = layer.listLayers()[0]
        print(l.dataSource)
    else:
        print(layer.datasetName)

get_filepath(r'C:\GIS\Temp\lyrs_remote_sde_few_features')

The summary of the tests I've done are below:

  • DBMS: SQL Server 2012
  • Enterprise geodatabase (aka SDE): 10.4.1
  • ArcGIS Desktop 10.4.1

The results:

  • For Python 32bit/64bit:
  • Same performance (fractions of seconds) to create a Layer object when the .lyr files refer to either local SQL Server database or when the script is run on the remote machine with the remote database.
  • Same performance (4-6 seconds) to create a Layer object when the .lyr files refer to remote SQL Server database and the script is run on the local machine.

The same performance was observed when running code:

  • in ArcMap Python window;
  • from cmd;
  • as a script tool in ArcMap;
  • using ArcGIS Pro Python 3.5 64bit
  • using arcpy.Describe(path).table.name

I have even written an ArcObjects script to see if this was an arcpy overhead:

from comtypes.client import GetModule, CreateObject
from snippets102 import GetStandaloneModules, InitStandalone

GetStandaloneModules()
InitStandalone()

esriCarto = GetModule(r"C:\Program Files (x86)\ArcGIS\Desktop10.4\com\esriCarto.olb")

layerFile = CreateObject(esriCarto.LayerFile,interface=esriCarto.ILayerFile)
lyrs = ["C:\GIS\Temp\lyrs_remote_sde_many_features\Parc.lyr",
        r"C:\GIS\Temp\lyrs_local_sde\Park_boundary.lyr",
        "C:\GIS\Temp\lyrs_remote_sde_few_features\Work.lyr"]

for lyr_file in lyrs:
    layerFile.Open(lyr_file)
    lyr = layerFile.Layer
    desc = lyr.QueryInterface(esriCarto.IFeatureLayer)
    print desc.DataSourceType
    #u'SDE Feature Class'
    print desc.FeatureClass.AliasName
    #sqlgdb.dbo.Park

And it takes the same amount of time (4-6 secs) to create a Layer object with ArcObjects. Again, it is always slow when accessing a remote database, but super fast when the Python code is run on the same machine where the database is hosted.

So, you either:

  • accept that it takes a lot of time to run the code and schedule it to run off work hours;
  • run your Python script on the machine where the SDE database is hosted;
  • pre-generate the .lyr files metadata and use it later for lookup.
  • Wow, this is exactly the answer I was looking for. So illuminating. Thank you. – Kristen G. Dec 14 '16 at 16:19
  • You are welcome, I was also eager to find the bottleneck. Do you think you would be able to host your Python script on the machine where the SDE database is hosted or at least on the same LAN? I think it is worth experimenting with this. – Alex Tereshenkov Dec 14 '16 at 17:17
  • It's something I will have to look into. I asked our application developer, and he doesn't host anything on that machine. I'll ask about the LAN today. He did suggest I make a copy of our SDE onto a dev box to test, but that's along the lines of what you've already done. Although it may be worth doing just to show mgmt how much faster things can be... Maybe it will open a conversation. I will definitely post updates as I continue to work on this! – Kristen G. Dec 15 '16 at 13:48
  • @KristenG., be careful when exporting the SDE feature classes into a file gdb - you will lose the information about the schema. Also, you will need to remap your lyr files to point to the file geodatabase instead, which again would require first creating a Layer object connecting to a remote db which is slow when done over the network... – Alex Tereshenkov Dec 15 '16 at 14:08
2

This doesn't answer your question as to why creating the Layer object is slow, but it's a possible solution for you to be able to find out which layer points to which dataset quickly. This is assuming that you need to perform this type of lookup often.

You could setup a weekly, monthly, or depending on however often new layer files are added to the directory, Python script that runs overnight (or whenever) and populates a new database table with the information you want.

You could just use a simple SQLite database.

So, you'd essentially use the same script you have now, but use it to populate the table. This way you'd only need to run it once in a while to refresh the data.

This would allow you to just query the database table whenever you need an answer.

SELECT datasetName, dataSourcePath, workspacePath, longName, name
FROM LayerFileTable
WHERE datasetName LIKE '%NYCMAP%' 

For the SELECT, I just used some properties from the Layer object documentation.

  • Interesting! So my current solution is to output an Excel file (which is searchable of course), although creating a database table as opposed to an Excel table is an interesting idea... Can you see any obvious benefit to using one over the other (Excel vs. database table)? – Kristen G. Dec 14 '16 at 14:44
  • 2
    @KristenG., I would go for a db table - much easier to execute queries to (everyone understands SQL). No locking issues you get when open the Excel file in many apps. And more functionality exposed via code - you could run nice aggregation and sorting. – Alex Tereshenkov Dec 14 '16 at 15:25
  • 2
    I was typing a reply, but Alex said it better than me. You could always write to both to see which you prefer, it shouldn't take any longer. But I'd say a db table is better fitting for this. If you don't have much experience with SQLite it might be fun too :) or whichever database you would use. – ianbroad Dec 14 '16 at 15:31
  • We've been trying to use more SQL in the dept, so this would be a great project. Also a great way to get around the locking issue (or accidentally editing the file and forgetting not to save) with Excel. Will report back on how this turns out. – Kristen G. Dec 14 '16 at 16:23
  • 1
    I just had a great idea (I think). Preferably, I'd love for the search to be an ArcToolbox script tool (which was always my aim, but had become unrealistic). I think if I had a database table, stored in our SDE, then I could write an ArcToolbox tool that would work super fast. And perhaps it could make use of SQL via the "Make Query Layer" arcpy function. I will give this a shot. The script to create the table would still have to be run after hours once every week or so... – Kristen G. Dec 14 '16 at 16:49
1

Just grasping at straws here.

Maybe you could go into ArcMap, uncheck the option to turn layers on when added to ArcMap, add all the layers to a map document, save said map document, then read all the layers in the map document with arcpy. That way you don't have to create the layer since it is in the map document. I have a map document with about 780 layers in it and it takes around 36.84 seconds to create a list of all the layers in the document when I run your script on only 47 layers files it takes 41.89 seconds.enter image description here enter image description here

  • 1
    thanks for your answer. Could you please copy/paste your code into your answer rather than a screenshot. It allows people to reuse your code easier. – Fezter Dec 7 '16 at 23:06
  • Yeah sorry, I'll have to do it later because I wasn't hurry to leave work after I answered the question – PMK Dec 7 '16 at 23:08
  • @Peter I tried to play around with your suggestion by adding a subset of 35 layers to a map document, and then running some of my code to get the datasetName. It works super fast, just like you said. However, the problem is that adding all of the layers to the map doc is now what will take 2-3 hours, since to do so you still need to use arcpy.mapping.layer to then use arcpy.mapping.AddLayer(). Adding layers has to be done programmatically since not only are there about 1500 layers, but they all exist within many sub-folders. It's just too much to do manually :/ – Kristen G. Dec 8 '16 at 19:24
  • I said 1,500 layers, but apparently what I meant was 2,957 layers! – Kristen G. Dec 12 '16 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.