Update Cursor keeps Crashing and Skipping

Question

I use a 3rd party software that exports the name of JPEG as I collect points, in this case street signs. I know the names stay static and are all unique but different features may have the same JPEG (which I think wont be a problem). So the issue I am having, as my update cursor is running it seems to crash and often times skip over fields (they do exist and are still being skipped over). When I run my codes again sometimes I will get this "Too many outputs to process". Here is my Code:

I am using Pycharm, I have some theories to why it keeps crashing, not enough memory, data too large, or the program is clunky when processing large data. If anyone has some suggestions or more efficient programs for processing python code for my large data sets that would be awesome.

Here is my Code:

import arcpy

from arcpy import management

arcpy.env.workspace = r'F:\Compton LTI Imagery\Imagery'

folders = arcpy.ListWorkspaces("*")

for folder in folders:

arcpy.env.workspace = folder

rasters = arcpy.ListRasters("*")

for raster in rasters:

    desc = arcpy.Describe(raster)

    input = r'C:\Users\ferna\Desktop\Compton_Signs\Compton_Signs.shp'

    management.AddField(input,"Direct_R", "TEXT", "","", 100)

    fields = ['Right_Img', 'Direct_R']

    with arcpy.da.UpdateCursor(input, fields)as cursor:

        for row in cursor:

            name = desc.baseName

            if row[0] == name:

                row[1] = folder + "\\" + raster

            cursor.updateRow(row)

            print raster + " " + "Directory_R Copied"

    management.AddField(input,"Direct_L", "TEXT", "","", 100)

    fields_2 =  ['Left_Img', 'Direct_L']

    with arcpy.da.UpdateCursor(input, fields_2)as cursor:

        for row in cursor:

            name = desc.baseName

            if row[0] == name:

                row[1] = folder + "\\" + raster

            cursor.updateRow(row)

            print raster + " " + "Directory_L Copied"

print "Operation Completed"

Answer 1

This:

import arcpy, traceback, os, sys
theFolder = r'C:\D drive from Toshiba\Peter_CD'
inputSHP = r'C:\FELIX_DATA\Compton_Signs.shp'
try:
    def showPyMessage():
        arcpy.AddMessage(str(time.ctime()) + " - " + message)
    aDictionary={}
    jpeg='.JPG'
    for root, dirs, files in os.walk(theFolder):
        for fname in files:
            uCase=fname.upper()
            if jpeg in uCase:
#            assuming jpeg names ARE unique
                aDictionary[uCase]=os.path.join(root, uCase)
    with arcpy.da.UpdateCursor(inputSHP, ['Right_Img','Left_Img',
                                          'Direct_R','Direct_L'])as cursor:
        for row in cursor:
            for m in range(2):
                aKey=(row[m]+jpeg).upper()
                try:
                    row[m+2]=aDictionary[aKey]
                except: row[m+2]=''
            cursor.updateRow(row)
except:
    message = "\n*** PYTHON ERRORS *** "; showPyMessage()
    message = "Python Traceback Info: " + traceback.format_tb(sys.exc_info()[2])[0]; showPyMessage()
    message = "Python Error Info: " +  str(sys.exc_type)+ ": " + str(sys.exc_value) + "\n"; showPyMessage()            

Takes 1 second to process 3800 images (all jpegs!) stored in hundreds of folders

Answer 2

I have a few suggestions:

• Where you have:

  with arcpy.da.UpdateCursor(input, fields)as cursor:

try putting a space before as:

    with arcpy.da.UpdateCursor(input, fields) as cursor:

• Where you have:

  with arcpy.da.UpdateCursor(input, fields_2)as cursor:

try deleting the row and cursor variables from your first cursor before reusing them, and adding the same space as before:

    del row,cursor
    with arcpy.da.UpdateCursor(input, fields_2) as cursor: