0

I am looping over a quite large feature class (~5million records) While looping using a SearchCursor, I insert new geometries in a FileGeodatabase FeatureClass using arcpy.da.InsertCursor. The inserted geometries are simple poygons (squares). The code works flaylessly for about 2k loops when ArcGIS suddenly crashes. Geometries fed to cursor.InsertRow() before crash are available in target FeatureClass. I felt like this was some kid of buffer issue so I sought for flushing commands but I could not find any in InsertCursor.

What can it be?

********** UPDATE *********

Adding script for clarification. My source FeatureClass is a Point FC with 2 text attributes: 'cellx' and 'celly' My destination FeatureClass isa Polygon FC with geometry plus one text attribute named 'gid'

hc = 0.00833333 # halfcell
# Open an InsertCursor toinsert new geometries
insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','gid'])
i = 1
with arcpy.da.SearchCursor("meteogrid_points_w_gid",['SHAPE@','celly','cellx']) as readCursor:
for row in readCursor:
    point = row[0].firstPoint
    x = point.X
    y = point.Y
    # Create a square polygon geometry (point(X, Y) around source point
    array = arcpy.Array([arcpy.Point(x+hc, y+hc),arcpy.Point(x+hc, y-hc),arcpy.Point(x-hc, y-hc),arcpy.Point(x-hc, y+hc)])
    polygon = arcpy.Polygon(array)
    # Create string to put in 'gid' attribute
    cellname = str(row[1])+"_"+str(row[2])
    # Log to screen for debugging
    print str(i)+" "+cellname
    i = i + 1
    insertCursor.insertRow([polygon,cellname])
  • 1
    If you open task manager and check the MEMORY of the process (arcmap?), do you see it growing and growing before it crashes? – KHibma Mar 16 '18 at 16:37
  • Memory usage is really limited. ArcMap is reported in Task Manager to be usin not even 80Mb (yes megabytes) and CPU time is ~15% (on a quad core - meaning not even a single core is saturated). If I launch the script in "blocks" (acting on the SearcCursor's where clause) the process ends perfectly. Might this be related to some kind of buffering on filesystem? I noticed that whan process has looped all expected features, I stays still for quite a while: as if it is writing to actual filegeodatabase the features it has cached somewhere else.. ..buw where? Not in memory: is still 80Mb also here. – mcfoi Mar 16 '18 at 17:11
  • Try Repair Geometry on source feature class – BERA Mar 16 '18 at 17:57
  • 1
    Should the Polygon have the same first and last point so it is a closed line ? – klewis Mar 16 '18 at 19:19
  • Please Edit the question to specify the exact version of ArcGIS. I've loaded tens of millions of rows, using point, line, and polygon topology scores of times in succession without an error (except when a dictionary cache exceeded 5.7Gb, and I needed to switch over to 64-bit Python), so it seems likely your exact data is involved in whatever the problem might be. If you can isolate the exact feature, then maybe we can help you figure out what is wrong. – Vince Mar 16 '18 at 21:14
0

After different tests I got to a "workaround": since the problem seemed to be the cursor, I just moved the creation of the cursor INSIDE THE LOOP.

hc = 0.00833333 # halfcell
insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','cellname'])
i = 1
with arcpy.da.SearchCursor("meteo_xy_map_new_full",['SHAPE@','celly','cellx'] ) as readCursor:
    if ((i%3000)==0):
        del insertCursor
        insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','cellname'])
        ...

This way the cursor gets recreated each 3000 inserts and stops reaching that mysterious limit causing ArcGIS to crash.

What I could see is that legacy ArcObject insert cursors had a flushing mechanism that got buried by the python wrappers... ..but its implemetation has some flaw or assumes some unclarified usage practice.

  • I had a project where I had to test the performance of loading 200 million features across several tables, one of which was 68m rows. I had no problems with 10.3.1, 10.4.0, or 10.4.1. I recently loaded 20m features with a DA InsertCursor using 10.4.1, 10.5.0, 10.5.1 and 10.6.0, with no difficulty. I'm not convinced there were any changes since DA cursors were released at 10.1. – Vince Mar 17 '18 at 16:16
  • You can accept your own answer with the checkbox – BERA Aug 31 '18 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.