1

I use a 3rd party software that exports the name of JPEG as I collect points, in this case street signs. I know the names stay static and are all unique but different features may have the same JPEG (which I think wont be a problem). So the issue I am having, as my update cursor is running it seems to crash and often times skip over fields (they do exist and are still being skipped over). When I run my codes again sometimes I will get this "Too many outputs to process". Here is my Code:

I am using Pycharm, I have some theories to why it keeps crashing, not enough memory, data too large, or the program is clunky when processing large data. If anyone has some suggestions or more efficient programs for processing python code for my large data sets that would be awesome.

Here is my Code:

import arcpy

from arcpy import management

arcpy.env.workspace = r'F:\Compton LTI Imagery\Imagery'

folders = arcpy.ListWorkspaces("*")

for folder in folders:

arcpy.env.workspace = folder

rasters = arcpy.ListRasters("*")

for raster in rasters:

    desc = arcpy.Describe(raster)

    input = r'C:\Users\ferna\Desktop\Compton_Signs\Compton_Signs.shp'

    management.AddField(input,"Direct_R", "TEXT", "","", 100)

    fields = ['Right_Img', 'Direct_R']

    with arcpy.da.UpdateCursor(input, fields)as cursor:

        for row in cursor:

            name = desc.baseName

            if row[0] == name:

                row[1] = folder + "\\" + raster

            cursor.updateRow(row)

            print raster + " " + "Directory_R Copied"

    management.AddField(input,"Direct_L", "TEXT", "","", 100)

    fields_2 =  ['Left_Img', 'Direct_L']

    with arcpy.da.UpdateCursor(input, fields_2)as cursor:

        for row in cursor:

            name = desc.baseName

            if row[0] == name:

                row[1] = folder + "\\" + raster

            cursor.updateRow(row)

            print raster + " " + "Directory_L Copied"

print "Operation Completed"

enter image description here

enter image description here

4
  • What exactly are you trying to accomplish? It looks like you are trying to populate a field--although, I am unclear what the purpose of the L and R image fields. You could populate the name by manipulating Direct_R. I noticed the indentation is off throughout your script. Also, avoid using the variable input as this is a reserved Python variable.
    – Aaron
    Dec 29, 2015 at 1:47
  • I have realised that your script is also pretty inefficient. Cursors are expensive resource-wise and I think it is best to collect all raster names in a list and use your conditionals on that list (e.g., if row[0] in rasters_in_this_folder:). By this way you will run cursor(s) only once and your fields will still be populated correctly. Also make sure the file name fields (DIRECT_* derivatives) can contain the string you are trying to write (i.e., field length of 100). If it still fails, you can collate all the information you need in a python dictionary and debug from there.
    – fatih_dur
    Dec 29, 2015 at 5:02
  • @Aaron what I am trying to accomplish that I want to get the root directory of the jpeg the feature point was used to get extracted from.
    – fernip0w
    Dec 29, 2015 at 17:33
  • @fatih_dur I realized with the suggestions below the method of going with my update cursor was very resource consuming. I have just started python coding in the recent months so I have a long way to go be writing logical and efficient codes. I appreciate your input.
    – fernip0w
    Dec 29, 2015 at 17:36

2 Answers 2

1

This:

import arcpy, traceback, os, sys
theFolder = r'C:\D drive from Toshiba\Peter_CD'
inputSHP = r'C:\FELIX_DATA\Compton_Signs.shp'
try:
    def showPyMessage():
        arcpy.AddMessage(str(time.ctime()) + " - " + message)
    aDictionary={}
    jpeg='.JPG'
    for root, dirs, files in os.walk(theFolder):
        for fname in files:
            uCase=fname.upper()
            if jpeg in uCase:
#            assuming jpeg names ARE unique
                aDictionary[uCase]=os.path.join(root, uCase)
    with arcpy.da.UpdateCursor(inputSHP, ['Right_Img','Left_Img',
                                          'Direct_R','Direct_L'])as cursor:
        for row in cursor:
            for m in range(2):
                aKey=(row[m]+jpeg).upper()
                try:
                    row[m+2]=aDictionary[aKey]
                except: row[m+2]=''
            cursor.updateRow(row)
except:
    message = "\n*** PYTHON ERRORS *** "; showPyMessage()
    message = "Python Traceback Info: " + traceback.format_tb(sys.exc_info()[2])[0]; showPyMessage()
    message = "Python Error Info: " +  str(sys.exc_type)+ ": " + str(sys.exc_value) + "\n"; showPyMessage()            

Takes 1 second to process 3800 images (all jpegs!) stored in hundreds of folders

1
  • I tried this, and it worked beautifully, I notice there was was some code I am unfamiliar with," aDictionary[uCase]=os.path.join(root, uCase)" is this a blend between what was imported from the OS, or SYS module? Thank you for your help.
    – fernip0w
    Dec 29, 2015 at 17:42
1

I have a few suggestions:

  • Where you have:

    with arcpy.da.UpdateCursor(input, fields)as cursor:

try putting a space before as:

    with arcpy.da.UpdateCursor(input, fields) as cursor:
  • Where you have:

    with arcpy.da.UpdateCursor(input, fields_2)as cursor:

try deleting the row and cursor variables from your first cursor before reusing them, and adding the same space as before:

    del row,cursor
    with arcpy.da.UpdateCursor(input, fields_2) as cursor:
1
  • I tried del, cursor but it keeps skipping over fields. the code mentioned above seemed to do the job. Is there any suggestions you can think of I can do to make this work?
    – fernip0w
    Dec 29, 2015 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.