Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

This is not a SDE question. File GDB.

Hi, I have a general question on looping through feature classes in a file geodatabase. I have (attached) a python script that adds a ton of fields and calculates them. It seems to run okay for 2-3 feature classes but by the time it hits the 4th or 5th one it often breaks down to Error 99999 or a dreaded schema lock error. Then if I comment out the first 2-3 fc's and run it just for 4 and 5 one at a time it runs fine.

Is there some way to 'clear' the loop at the end or should I MakeTableView do everything, export the table, then Delete the TableView? Should I be using environments too instead of using the long paths?

I don't have many people look at my code and have no real training so if there are any very very obvious things I'm doing wrong please let me know.

import arcpy, time, datetime, csv
from arcpy import env
env.overwriteOutput = True

calcfiles = [
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/nyc_0250m_census",
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/nyc_1000m_census",
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/phi_0250m_census",
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/phi_1000m_census",
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/saj_0250m_census",
"E:/SpiderOak/projects/streetview/pedestrian_safety/processing/tables/census_joins.gdb/saj_1000m_census"
]

prefld = ["hkq",
"hk1",
"hkq",
"hk1",
"hkq",
"hk1"
]

print 'Calc Apportioned Census Loop started at this time: ' + time.strftime('%c') 

for cfname, prefieldname in zip(calcfiles, prefld):   
      cenvar = [
            "totpop",         #'!SET001001!*!pctorgarea!', #Total Population

            "sbatot",         #'!SET005001!*!pctorgarea!', #Total Population:

            "sbamal",         #'!SET005002!*!pctorgarea!',   #Male:
            "sbam05",         #'!SET005003!*!pctorgarea!',         # Under 5 Years
            "sbam09",         #'!SET005004!*!pctorgarea!',         #5 to 9 Years
            "sbam14",         #'!SET005005!*!pctorgarea!',         #10 to 14 Years
            "sbam17",         #'!SET005006!*!pctorgarea!',         #15 to 17 Years
            "sbam24",         #'!SET005007!*!pctorgarea!',         #18 to 24 Years
            "sbam34",         #'!SET005008!*!pctorgarea!',         #25 to 34 Years
            "sbam44",         #'!SET005009!*!pctorgarea!',         #35 to 44 Years
            "sbam54",         #'!SET005010!*!pctorgarea!',         #45 to 54 Years
            "sbam64",         #'!SET005011!*!pctorgarea!',         #55 to 64 Years
            "sbam74",         #'!SET005012!*!pctorgarea!',         #65 to 74 Years
            "sbam84",         #'!SET005013!*!pctorgarea!',         #75 to 84 Years
            "sbamel",         #'!SET005014!*!pctorgarea!',         #85 Years and over

            "sbafem",         #'!SET005015!*!pctorgarea!',   #Female:
            "sbaf05",         #'!SET005016!*!pctorgarea!',         #Under 5 Years
            "sbaf09",         #'!SET005017!*!pctorgarea!',         #5 to 9 Years
            "sbaf14",         #'!SET005018!*!pctorgarea!',         #10 to 14 Years
            "sbaf17",         #'!SET005019!*!pctorgarea!',         #15 to 17 Years
            "sbaf24",         #'!SET005020!*!pctorgarea!',         #18 to 24 Years
            "sbaf34",         #'!SET005021!*!pctorgarea!',         #25 to 34 Years
            "sbaf44",         #'!SET005022!*!pctorgarea!',         #35 to 44 Years
            "sbaf54",         #'!SET005023!*!pctorgarea!',         #45 to 54 Years
            "sbaf64",         #'!SET005024!*!pctorgarea!',         #55 to 64 Years
            "sbaf74",         #'!SET005025!*!pctorgarea!',         #65 to 74 Years
            "sbaf84",         #'!SET005026!*!pctorgarea!',         #75 to 84 Years
            "sbafel",         #'!SET005027!*!pctorgarea!',         #85 Years and over

            "ractot",         #'!SET013001!*!pctorgarea!',   #Total Population:
            "racwht",         #'!SET013002!*!pctorgarea!',      #White Alone
            "racblk",         #'!SET013003!*!pctorgarea!',      #Black or African American Alone
            "racnat",         #'!SET013004!*!pctorgarea!',      #American Indian and Alaska Native Alone
            "racasn",         #'!SET013005!*!pctorgarea!',      #Asian Alone
            "racpac",         #'!SET013006!*!pctorgarea!',      #Native Hawaiian and Other Pacific Islander Alone
            "racotr",         #'!SET013007!*!pctorgarea!',      #Some Other Race Alone
            "ractwo",         #'!SET013008!*!pctorgarea!',      #Two or More races

            "histot",      #'!SET014001*!pctorgarea!',    #Total Population
            "hisnoh",      #'!SET014002*!pctorgarea!',      #Not Hispanic or Latino:
            "hisnwh",      #'!SET014003*!pctorgarea!',          #White Alone
            "hisnbl",      #'!SET014004*!pctorgarea!',          #Black or African American Alone
            "hisnna",      #'!SET014005*!pctorgarea!',          #American Indian and Alaska Native Alone
            "hisnas",      #'!SET014006*!pctorgarea!',          #Asian Alone
            "hisnpa",      #'!SET014007*!pctorgarea!',          #Native Hawaiian and Other Pacific Islander Alone
            "hisntr",      #'!SET014008*!pctorgarea!',          #Some Other Race Alone
            "hisntw",      #'!SET014009*!pctorgarea!',          #Two or More races
            "hisyes",      #'!SET014010*!pctorgarea!',       #Hispanic or Latino:
            "hisywh",      #'!SET014011*!pctorgarea!',          #White Alone
            "hisybl",      #'!SET014012*!pctorgarea!',          #Black or African American Alone
            "hisyna",      #'!SET014013*!pctorgarea!',          #American Indian and Alaska Native Alone
            "hisyas",      #'!SET014014*!pctorgarea!',          #Asian Alone
            "hisypa",      #'!SET014015*!pctorgarea!',          #Native Hawaiian and Other Pacific Islander Alone
            "hisytr",      #'!SET014016*!pctorgarea!',          #Some Other Race Alone
            "hisytw",      #!SET014017*!pctorgarea!',          #Two or More races

            "edutot",         #'!SET025001*!pctorgarea!',   #Population 25 Years and over:
            "edulhs",         #'!SET025002*!pctorgarea!',      #Less Than High School
            "eduhsg",         #'!SET025003*!pctorgarea!',      #High School Graduate (includes equivalency)
            "edusoc",         #'!SET025004*!pctorgarea!',      #Some college
            "edubac",         #'!SET025005*!pctorgarea!',      #Bachelor's degree
            "edumas",         #'!SET025006*!pctorgarea!',      #Master's degree
            "eduprf",         #'!SET025007*!pctorgarea!',      #Professional school degree
            "edudoc",         #'!SET025008*!pctorgarea!',      #Doctorate degree

            "emptot",         #'!SET037001*!pctorgarea!',   #Civilian Population In Labor Force 16 Years And Over:
            "empemp",         #'!SET037002*!pctorgarea!',      #Employed
            "empune",         #'!SET037003*!pctorgarea!',      #Unemployed

            "medhhi",         #'!SET057001!',                     #Median household income (In 2010 Inflation Adjusted Dollars)

            "pastot",         #'!SET080001*!pctorgarea!',   #Households:
            "paswit",         #'!SET080002*!pctorgarea!',      #With public assistance income
            "pasnpu",         #'!SET080003*!pctorgarea!',      #No public assistance income

            "housun",         #'!SET093001*!pctorgarea!',    #Housing units

            "tentot",         #'!SET094001*!pctorgarea!',   #Occupied Housing Units:
            "tenown",         #'!SET094002*!pctorgarea!',      #Owner Occupied
            "tenren",         #'!SET094003*!pctorgarea!',      #Renter Occupied

            "occtot",         #'!SET095001*!pctorgarea!',   #Housing units:
            "occocc",         #'!SET095002*!pctorgarea!',     #Occupied
            "occvac",         #'!SET095003*!pctorgarea!',     #Vacant

            "grttot",         #'!SET102001*!pctorgarea!',   #Renter-occupied housing units with cash rent:
            "grtl03",         #'!SET102002*!pctorgarea!',      #Less than $300
            "grtl06",         #'!SET102003*!pctorgarea!',      #$300 to $599
            "grtl08",         #'!SET102004*!pctorgarea!',      #$600 to $799
            "grtl10",         #'!SET102005*!pctorgarea!',      #$800 to $999
            "grtl12",         #'!SET102006*!pctorgarea!',      #$1,000 to $1,249
            "grtl15",         #'!SET102007*!pctorgarea!',      #$1,250 to $1,499
            "grtl20",         #'!SET102008*!pctorgarea!',      #$1,500 to $1,999
            "grta20",         #'!SET102009*!pctorgarea!',      #$2,000 or More

            "rntmed",         #'!SET104001!',   #Median Gross Rent

            "povtot",         #'!SET113001!*!pctorgarea!',      #   Families:
            "povpov",         #'!SET113002!*!pctorgarea!',      #      Income in 2010 below poverty level:
            "povcwc",         #'!SET113003!*!pctorgarea!',      #         Married Couple Family: With Related Child Living  Bellow Poverty Level
            "povcnc",         #'!SET113004!*!pctorgarea!',      #         Married Couple Family: No related children under 18 Years
            "povmal",         #'!SET113005!*!pctorgarea!',      #         Male Householder, no wife present:
            "povmwc",         #'!SET113006!*!pctorgarea!',      #            With related children under 18 Years
            "povmnc",         #'!SET113007!*!pctorgarea!',      #            No related children under 18 Years
            "povfem",         #'!SET113008!*!pctorgarea!',      #         Female Householder, no husband present:
            "povfwc",         #'!SET113009!*!pctorgarea!',      #            With related children under 18 Years
            "povnoc",         #'!SET113010!*!pctorgarea!',      #            No related children under 18 Years
            "povabv",         #'!SET113011!*!pctorgarea!',      #      Income In 2010 at or above poverty level

            "trnw16",         #'!SET128001!*!pctorgarea!',      #   Workers 16 Years and over:
            "trncar",         #'!SET128002!*!pctorgarea!',      #      Car, truck, or van
            "trntax",         #'!SET128003!*!pctorgarea!',      #      Public transportation (Includes Taxicab)
            "trnmot",         #'!SET128004!*!pctorgarea!',      #      Motorcycle
            "trnbik",         #'!SET128005!*!pctorgarea!',      #      Bicycle
            "trnwlk",         #'!SET128006!*!pctorgarea!',      #      Walked
            "trnotr",         #'!SET128007!*!pctorgarea!',      #      Other means
            "trnhom",         #'!SET128008!*!pctorgarea!',      #      Worked at home

            "timw16",         #'!SET129001!*!pctorgarea!',      #   Workers 16 Years and over:
            "timwka",         #'!SET129002!*!pctorgarea!',      #      Did not work at home:
            "timu10",         #'!SET129003!*!pctorgarea!',      #         Less than 10 minutes
            "timu20",         #'!SET129004!*!pctorgarea!',      #         10 to 19 minutes
            "timu30",         #'!SET129005!*!pctorgarea!',      #         20 to 29 minutes
            "timu40",         #'!SET129006!*!pctorgarea!',      #         30 to 39 minutes
            "timu60",         #'!SET129007!*!pctorgarea!',      #         40 to 59 minutes
            "timu90",         #'!SET129008!*!pctorgarea!',      #         60 to 89 minutes
            "tim90p",         #'!SET129009!*!pctorgarea!',      #         90 or More minutes
            "timhom",         #'!SET129010!*!pctorgarea!',      #      Worked at home

            "nattot",         #'!SET133001!*!pctorgarea!',      #   Total Population:
            "natnat",         #'!SET133002!*!pctorgarea!',      #      Native Born
            "natfor",         #'!SET133003!*!pctorgarea!',      #      Foreign Born:
            "natntz",         #'!SET133004!*!pctorgarea!',      #         Naturalized Citizen
            "natnoc",         #'!SET133005!*!pctorgarea!',      #         Not a Citizen

            "ownocc",         #'!SET142001!*!pctorgarea!',      #   Owner-occupied housing units
            ]

      varexp = [
            '!SET001001!*!pctorgarea!', #Total Population

            '!SET005001!*!pctorgarea!', #Total Population:

            '!SET005002!*!pctorgarea!',   #Male:
            '!SET005003!*!pctorgarea!',         # Under 5 Years
            '!SET005004!*!pctorgarea!',         #5 to 9 Years
            '!SET005005!*!pctorgarea!',         #10 to 14 Years
            '!SET005006!*!pctorgarea!',         #15 to 17 Years
            '!SET005007!*!pctorgarea!',         #18 to 24 Years
            '!SET005008!*!pctorgarea!',         #25 to 34 Years
            '!SET005009!*!pctorgarea!',         #35 to 44 Years
            '!SET005010!*!pctorgarea!',         #45 to 54 Years
            '!SET005011!*!pctorgarea!',         #55 to 64 Years
            '!SET005012!*!pctorgarea!',         #65 to 74 Years
            '!SET005013!*!pctorgarea!',         #75 to 84 Years
            '!SET005014!*!pctorgarea!',         #85 Years and over

            '!SET005015!*!pctorgarea!',   #Female:
            '!SET005016!*!pctorgarea!',         #Under 5 Years
            '!SET005017!*!pctorgarea!',         #5 to 9 Years
            '!SET005018!*!pctorgarea!',         #10 to 14 Years
            '!SET005019!*!pctorgarea!',         #15 to 17 Years
            '!SET005020!*!pctorgarea!',         #18 to 24 Years
            '!SET005021!*!pctorgarea!',         #25 to 34 Years
            '!SET005022!*!pctorgarea!',         #35 to 44 Years
            '!SET005023!*!pctorgarea!',         #45 to 54 Years
            '!SET005024!*!pctorgarea!',         #55 to 64 Years
            '!SET005025!*!pctorgarea!',         #65 to 74 Years
            '!SET005026!*!pctorgarea!',         #75 to 84 Years
            '!SET005027!*!pctorgarea!',         #85 Years and over

            '!SET013001!*!pctorgarea!',   #Total Population:
            '!SET013002!*!pctorgarea!',      #White Alone
            '!SET013003!*!pctorgarea!',      #Black or African American Alone
            '!SET013004!*!pctorgarea!',      #American Indian and Alaska Native Alone
            '!SET013005!*!pctorgarea!',      #Asian Alone
            '!SET013006!*!pctorgarea!',      #Native Hawaiian and Other Pacific Islander Alone
            '!SET013007!*!pctorgarea!',      #Some Other Race Alone
            '!SET013008!*!pctorgarea!',      #Two or More races

            '!SET014001!*!pctorgarea!',    #Total Population
            '!SET014002!*!pctorgarea!',      #Not Hispanic or Latino:
            '!SET014003!*!pctorgarea!',          #White Alone
            '!SET014004!*!pctorgarea!',          #Black or African American Alone
            '!SET014005!*!pctorgarea!',          #American Indian and Alaska Native Alone
            '!SET014006!*!pctorgarea!',          #Asian Alone
            '!SET014007!*!pctorgarea!',          #Native Hawaiian and Other Pacific Islander Alone
            '!SET014008!*!pctorgarea!',          #Some Other Race Alone
            '!SET014009!*!pctorgarea!',          #Two or More races
            '!SET014010!*!pctorgarea!',       #Hispanic or Latino:
            '!SET014011!*!pctorgarea!',          #White Alone
            '!SET014012!*!pctorgarea!',          #Black or African American Alone
            '!SET014013!*!pctorgarea!',          #American Indian and Alaska Native Alone
            '!SET014014!*!pctorgarea!',          #Asian Alone
            '!SET014015!*!pctorgarea!',          #Native Hawaiian and Other Pacific Islander Alone
            '!SET014016!*!pctorgarea!',          #Some Other Race Alone
            '!SET014017!*!pctorgarea!',          #Two or More races

            '!SET025001!*!pctorgarea!',   #Population 25 Years and over:
            '!SET025002!*!pctorgarea!',      #Less Than High School
            '!SET025003!*!pctorgarea!',      #High School Graduate (includes equivalency)
            '!SET025004!*!pctorgarea!',      #Some college
            '!SET025005!*!pctorgarea!',      #Bachelor's degree
            '!SET025006!*!pctorgarea!',      #Master's degree
            '!SET025007!*!pctorgarea!',      #Professional school degree
            '!SET025008!*!pctorgarea!',      #Doctorate degree

            '!SET037001!*!pctorgarea!',   #Civilian Population In Labor Force 16 Years And Over:
            '!SET037002!*!pctorgarea!',      #Employed
            '!SET037003!*!pctorgarea!',      #Unemployed

            '!SET057001!',                      #Median household income (In 2010 Inflation Adjusted Dollars)

            '!SET080001!*!pctorgarea!',   #Households:
            '!SET080002!*!pctorgarea!',      #With public assistance income
            '!SET080003!*!pctorgarea!',      #No public assistance income

            '!SET093001!*!pctorgarea!',    #Housing units

            '!SET094001!*!pctorgarea!',   #Occupied Housing Units:
            '!SET094002!*!pctorgarea!',      #Owner Occupied
            '!SET094003!*!pctorgarea!',      #Renter Occupied

            '!SET095001!*!pctorgarea!',   #Housing units:
            '!SET095002!*!pctorgarea!',     #Occupied
            '!SET095003!*!pctorgarea!',     #Vacant

            '!SET102001!*!pctorgarea!',   #Renter-occupied housing units with cash rent:
            '!SET102002!*!pctorgarea!',      #Less than $300
            '!SET102003!*!pctorgarea!',      #$300 to $599
            '!SET102004!*!pctorgarea!',      #$600 to $799
            '!SET102005!*!pctorgarea!',      #$800 to $999
            '!SET102006!*!pctorgarea!',      #$1,000 to $1,249
            '!SET102007!*!pctorgarea!',      #$1,250 to $1,499
            '!SET102008!*!pctorgarea!',      #$1,500 to $1,999
            '!SET102009!*!pctorgarea!',      #$2,000 or More

            '!SET104001!',                      #Median Gross Rent

            '!SET113001!*!pctorgarea!',      #   Families:
            '!SET113002!*!pctorgarea!',      #      Income in 2010 below poverty level:
            '!SET113003!*!pctorgarea!',      #         Married Couple Family: With Related Child Living  Bellow Poverty Level
            '!SET113004!*!pctorgarea!',      #         Married Couple Family: No related children under 18 Years
            '!SET113005!*!pctorgarea!',      #         Male Householder, no wife present:
            '!SET113006!*!pctorgarea!',      #            With related children under 18 Years
            '!SET113007!*!pctorgarea!',      #            No related children under 18 Years
            '!SET113008!*!pctorgarea!',      #         Female Householder, no husband present:
            '!SET113009!*!pctorgarea!',      #            With related children under 18 Years
            '!SET113010!*!pctorgarea!',      #            No related children under 18 Years
            '!SET113011!*!pctorgarea!',      #      Income In 2010 at or above poverty level

            '!SET128001!*!pctorgarea!',      #   Workers 16 Years and over:
            '!SET128002!*!pctorgarea!',      #      Car, truck, or van
            '!SET128003!*!pctorgarea!',      #      Public transportation (Includes Taxicab)
            '!SET128004!*!pctorgarea!',      #      Motorcycle
            '!SET128005!*!pctorgarea!',      #      Bicycle
            '!SET128006!*!pctorgarea!',      #      Walked
            '!SET128007!*!pctorgarea!',      #      Other means
            '!SET128008!*!pctorgarea!',      #      Worked at home

            '!SET129001!*!pctorgarea!',      #   Workers 16 Years and over:
            '!SET129002!*!pctorgarea!',      #      Did not work at home:
            '!SET129003!*!pctorgarea!',      #         Less than 10 minutes
            '!SET129004!*!pctorgarea!',      #         10 to 19 minutes
            '!SET129005!*!pctorgarea!',      #         20 to 29 minutes
            '!SET129006!*!pctorgarea!',      #         30 to 39 minutes
            '!SET129007!*!pctorgarea!',      #         40 to 59 minutes
            '!SET129008!*!pctorgarea!',      #         60 to 89 minutes
            '!SET129009!*!pctorgarea!',      #         90 or More minutes
            '!SET129010!*!pctorgarea!',      #      Worked at home

            '!SET133001!*!pctorgarea!',      #   Total Population:
            '!SET133002!*!pctorgarea!',      #      Native Born
            '!SET133003!*!pctorgarea!',      #      Foreign Born:
            '!SET133004!*!pctorgarea!',      #         Naturalized Citizen
            '!SET133005!*!pctorgarea!',      #         Not a Citizen

            '!SET142001!*!pctorgarea!',      #   Owner-occupied housing units
            ]

      for fname, expre in zip(cenvar, varexp):
            try:
                #arcpy.AddField_management(cfname, fname, "FLOAT")
                #arcpy.CalculateField_management(cfname, fname, expre, "PYTHON")
                arcpy.AddField_management(cfname, prefieldname+fname, "FLOAT")
                arcpy.CalculateField_management(cfname, prefieldname+fname, expre, "PYTHON")
                  # this is with the prefieldname gonna add that later, don't neeed now
                print 'Loop ' + prefieldname + " - field: " + fname + ' done at this time: ' + time.strftime('%c') 
            except:
                print 'Script for ' + cfname + ' screwed up at this time: ' + time.strftime('%c') 

print 'Script ended at this time: ' + time.strftime('%c') 
share|improve this question
    
Perhaps something else to ask would be, should I turn this whole thing into a function and then loop through my data to the function, AddCalcCensus(calcfiles,prefld) –  GIS Danny Mar 22 '13 at 17:43
1  
I wonder if the tools are running async, so while it is wrapping up calculatefield for one feature class, it starts on addfield for another. Try doing -only- your AddField operations and see if that works (since AddField is quicker). I would definitely go the function route though. –  blord-castillo Mar 22 '13 at 17:53
    
Is the rest of your Error 99999 message about "General Function Failure"? –  PolyGeo Mar 23 '13 at 3:30
1  
Could this be a performance issue? I've had older pc's that made Arcmap cranking with lots of 99999's. Check this post . MakeFeatureLayer and "in_memory" workspace can help streamline. –  gm70560 Mar 26 '13 at 23:36
    
@blord-castillo I think your suggestion helps. I tried a version of this in another script (with fewer variables added/calced) and it seemed to not lock up. I will try and modify this code so there's one loop of adding fields and one of calc'ing them. Thanks! I'm pretty new let me know if I didn't vote your comment correctly. I'll also check out in_memory as some people have mentioned that for other issues. –  GIS Danny Mar 28 '13 at 1:39
add comment

1 Answer

up vote 1 down vote accepted

I'm a bit late to this conversation, but a general comment on the non-arcpy python structure is that you instantiate cenvar and varexp each time you iterate w/in your for cfname, prefieldname loop, which is wasteful. Do it once prior to the loop if it’s not going to change. Also, use itertools.izip instead of zip in general. Your lists aren’t killer long, but using iterator equivalents of functions is generally good for speed/memory management.

share|improve this answer
    
Oh, so all of that stuff should come before the loop? Thanks. I have been using the comments above and doing the AddField and Calc fields not in the same loop and that helps for adding and calcing many fields. Also, running it as a python file outside of ArcGIS seems to help from errors that may or may not be schema lock related. –  GIS Danny Jun 6 '13 at 20:49
    
Those comments are good, but are different. I would use them all. I was a little confused by the arcpy part of the conversation because the AddField() and CalculateField() don't usually leave a lock behind (like a MakeFeatureLayer() would), so I wasn't sure how you were ending up with that condition (or thought you were). It would be interesting to know if you got more back from the messaging (like @PolyGeo asked). –  Roland Jun 6 '13 at 22:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.