2

Trying to bulk update versioned Feature Classes containing in excess of 30,000 records is causing issue due to the Reconcile/Post process taking too long and ending up with the notice saying that the target version has changed and user must Reconcile and Post again (and whole process repeats).

I have attempted to write a little script that will create Subsets (containing about 5000 records) of my original selection and run the Calculate Field, Reconcile, and Post functions on each subset rather than the entire selection.

I can get the list of ObjectIDs of the entire original selection using desc.FIDSet, however all the documentation points towards FIDSet being Read Only. In the script I have generated a selection query to use in the SelectLayerByAttribute but I'd like to know if there is a way to just recreate the FIDSet (or alternative term if not FIDSet) rather than passing through the Select query.

Or is there a better way to achieve what I'm attempting?

desc = arcpy.Describe(FeatureLayer) # Enables listing selected records by ObjectID through FIDSet

if desc.FIDSet != '': # Check if there are selected records
    FullSel = desc.FIDSet.split(";") # Create list of ObjectIDs of selected records
    print "Number of selected features: {}".format(len(desc.FIDSet.split(";")))

    while FullSel: # Keep going while there are still records to process
        SubsetSel = [] # Create SubsetSel list for groups of selected features
        for Sel in FullSel:
            if len(SubsetSel) < MaxLoopFeatures: # SubsetSel list will only have maximum number of features from MaxLoopFeatures
                SubsetSel.append(Sel) 

        ##======================================
        ## Is there a better way to do this??
        ##======================================
        query = 'OBJECTID IN (' + ','.join(map(str, SubsetSel)) + ')' # Create query for SelectlayerByAttribute to reselect features using SubsetSel
        ##print query
        arcpy.SelectLayerByAttribute_management(FeatureLayer, "NEW_SELECTION", query) # Reselect features
        ##======================================
        ##======================================

        desc2 = arcpy.Describe(FeatureLayer)
        print "Selection Set: {}".format(desc2.FIDSet)
        print "Number of selected features: {}".format(len(desc2.FIDSet.split(";")))

        # Run intended process here
        #arcpy.CalculateField_management()
        #arcpy.ReconcileVersion_management()

        FullSel = [x for x in FullSel if x not in SubsetSel] # Remove processed features from FullSel and repeat on remaining features
else:
    print "Nothing Selected"
  • if this is about update what's wrong with arcpy.da.UpdateCursor with sql_clause – FelixIP Jan 28 '16 at 0:50
  • Are you suggesting to use the UpdateCursor rather than Select and Calculate Field for each group of features? – Midavalo Jan 28 '16 at 0:52
  • 2
    Exactly. Field calculator is hopeless compare to da.UpdateCursor – FelixIP Jan 28 '16 at 0:54
  • 1
    @FelixIP, sounds like an answer! – artwork21 Jan 28 '16 at 1:54
  • It does indeed. I'm trying it out now. – Midavalo Jan 28 '16 at 2:13
2

Both select by attribute and field calculator are very slow. Moreover I found that calculator expressions aren’t easy to build.

This is why I am suggesting to use da.UpdateCursor with sql_clause to split dataset in groups and to modify field(s) values.

BTW FIDset changes automatically, no need to re-describe FeatureLayer every time

  • re: "BTW FIDset changes automatically" I guess you're meaning where I've put desc2 = arcpy.Describe(FeatureLayer) towards the end? This was just for debug / proof of concept. Will be gone when I get it all working. – Midavalo Jan 28 '16 at 2:16
  • My experience this morning with using da.UpdateCursor seems to be different from yours. Took just under 4 minutes to update 53000 records using my Select and Calculate Field method as posted above, however my attempt at an Update Cursor was still updating the first 5000 records when I killed it at 10 minutes. I imagine I've done something wrong, but wonder if it's the fact that I need to use arcpy.da.Editor to get the UpdateCursor to work that is slowing everything down. I'll keep trying, but may end up with another question. – Midavalo Jan 28 '16 at 21:56
  • I don't even know how to use arcpy.da.Editor. Have a look at gis.stackexchange.com/questions/177923/… – FelixIP Jan 28 '16 at 22:05
  • It seems that because my dataset is versioned the UpdateCursor needs to be in an edit session. When I tried it on a FileGeodatabase it worked without editor and very quickly. Unfortunately that won't be a solution. Thanks I'll take a look at that link. – Midavalo Jan 28 '16 at 22:10
0

Stumbled across the answer to this while looking for something else yesterday.

The Layer object has two methods getSelectionSet() and setSelectionSet().

getSelectionSet() is similar to the Describe.FIDSet except that it returns a List of ObjectIDs rather than FIDSet's string of ObjectIDs.

setSelectionSet() will pass a list of ObjectIDs to the layer to select them. It works much faster than the query = ... and arcpy.SelectLayerByAttribute_management code in my question above.

If I remove the Describe and FIDSet and replace with

lyr = arcpy.mapping.Layer(FeatureLayer)
FullSel = lyr.getSelectionSet()

if FullSel:

and then replace the query and select with

lyr.setSelectionSet("NEW", SubsetSel)

my test run takes about 24 seconds rather than almost 2 minutes.

lyr = arcpy.mapping.Layer(FeatureLayer) 
FullSel = lyr.getSelectionSet() # Enables listing selected records by ObjectID through SelectionSet

if FullSel: # Check if there are selected records
    print "Number of selected features: {}".format(len(FullSel))

    while FullSel: # Keep going while there are still records to process
        SubsetSel = [] # Create SubsetSel list for groups of selected features
        for Sel in FullSel:
            if len(SubsetSel) < MaxLoopFeatures: # SubsetSel list will only have maximum number of features from MaxLoopFeatures
                SubsetSel.append(Sel) 

        ##======================================
        ## Better way to do it!
        ##====================================== 

        lyr.setSelectionSet("NEW", SubsetSel)

        ##======================================
        ##======================================

        # Run intended process here
        #arcpy.CalculateField_management()
        #arcpy.ReconcileVersion_management()

        FullSel = [x for x in FullSel if x not in SubsetSel] # Remove processed features from FullSel and repeat on remaining features
else:
    print "Nothing Selected"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.