0

I have a large hosted feature layer on ArcGIS Online of 52,000 points. I have been building the feature layer over time by simply using the Append tool in ArcGIS Pro. Now I need to update some text and date values in these records, and I am finding that the computer takes over an hour to do so using arcpy.da.UpdateCursor. Is there a better way to do this? Do ArcGIS API for Python or ArcREST offer improved performance over an UpdateCursor? I have not used these libraries yet.

Here is an example of the simplest UpdateCursor that took over an hour - but I have many longer ones I need to do updating 10 or more fields for tens of thousands of records.

with arcpy.da.UpdateCursor(Layer,['date_approved', 'date_cleared']) as UC:
    for row in UC:
        if row[0] is not None:
            row[1] = row[0]
            UC.updateRow(row)
1
  • 4
    I regularly load millions of records to Enterprise geodatabases using ArcPy DA InsertCursor in minutes. Updates to thousands of rows are subsecond. It's not the tool you use to access the web-based dataset, it's the fact that it's a web-based dataset that generates the delay. You really ought to use a where_clause in the cursor declaration, so you're not forced to do multi-node transfer for records that won't be updated. – Vince Nov 10 '20 at 1:39
2

As your code is now, you're doing a row-by-row update. This means the code needs to send 52,000 requests and get 52,000 responses -- at a minimum. As Vince points out in the comment, this is a quick operation when you're in the same network to an Enterprise Geodatabase. But over the web, through a service and ultimately into the data adds overhead. Per Vince, you could cut down the number of requests and only operate against rows that need to be updated by making use of a where_clause in the cursor.

Per your question, will another technology be faster? Perhaps, but from my experience, I think they'll all take the same amount of time. You'd have to profile and watch the web requests to know for sure. We do know for sure, based on your code that the UpdateCursor will be a row-by-row update. I'm almost positive the Python API (and ArcREST, its basically the same thing) will do a feature-by-feature update. If you use Fiddler you can watch the requests and know for sure. Basically you want to watch and see if multiple features are sent in a single request, or if all feature updates are sent in a single. If all of the "out of the box" APIs send singular updates, you'd need to write your own code to do a bulk append.

Those are the only 2 ways to speed this up:

  • Only operate against items that need updating (where_clause)
  • Send 1 (or very few) requests that do a bulk update

Both these ideas essentially cut down the number of web transactions.

1
  • 1
    Thank you! I added a where clause and it worked in seconds on a test run. Really appreciate how clear your answer was. – Dylan Warburg Nov 10 '20 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.