0

I am looping over a quite large feature class (~5million records) While looping using a SearchCursor, I insert new geometries in a FileGeodatabase FeatureClass using arcpy.da.InsertCursor. The inserted geometries are simple poygons (squares). The code works flaylessly for about 2k loops when ArcGIS suddenly crashes. Geometries fed to cursor.InsertRow() before crash are available in target FeatureClass. I felt like this was some kid of buffer issue so I sought for flushing commands but I could not find any in InsertCursor.

What can it be?

********** UPDATE *********

Adding script for clarification. My source FeatureClass is a Point FC with 2 text attributes: 'cellx' and 'celly' My destination FeatureClass isa Polygon FC with geometry plus one text attribute named 'gid'

hc = 0.00833333 # halfcell
# Open an InsertCursor toinsert new geometries
insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','gid'])
i = 1
with arcpy.da.SearchCursor("meteogrid_points_w_gid",['SHAPE@','celly','cellx']) as readCursor:
for row in readCursor:
    point = row[0].firstPoint
    x = point.X
    y = point.Y
    # Create a square polygon geometry (point(X, Y) around source point
    array = arcpy.Array([arcpy.Point(x+hc, y+hc),arcpy.Point(x+hc, y-hc),arcpy.Point(x-hc, y-hc),arcpy.Point(x-hc, y+hc)])
    polygon = arcpy.Polygon(array)
    # Create string to put in 'gid' attribute
    cellname = str(row[1])+"_"+str(row[2])
    # Log to screen for debugging
    print str(i)+" "+cellname
    i = i + 1
    insertCursor.insertRow([polygon,cellname])
Improve this question
8
  • 1
    If you open task manager and check the MEMORY of the process (arcmap?), do you see it growing and growing before it crashes?
    – KHibma
    Commented Mar 16, 2018 at 16:37
  • Memory usage is really limited. ArcMap is reported in Task Manager to be usin not even 80Mb (yes megabytes) and CPU time is ~15% (on a quad core - meaning not even a single core is saturated). If I launch the script in "blocks" (acting on the SearcCursor's where clause) the process ends perfectly. Might this be related to some kind of buffering on filesystem? I noticed that whan process has looped all expected features, I stays still for quite a while: as if it is writing to actual filegeodatabase the features it has cached somewhere else.. ..buw where? Not in memory: is still 80Mb also here.
    – mcfoi
    Commented Mar 16, 2018 at 17:11
  • Try Repair Geometry on source feature class
    – Bera
    Commented Mar 16, 2018 at 17:57
  • 1
    Should the Polygon have the same first and last point so it is a closed line ?
    – klewis
    Commented Mar 16, 2018 at 19:19
  • Please Edit the question to specify the exact version of ArcGIS. I've loaded tens of millions of rows, using point, line, and polygon topology scores of times in succession without an error (except when a dictionary cache exceeded 5.7Gb, and I needed to switch over to 64-bit Python), so it seems likely your exact data is involved in whatever the problem might be. If you can isolate the exact feature, then maybe we can help you figure out what is wrong.
    – Vince
    Commented Mar 16, 2018 at 21:14

1 Answer 1

Reset to default
1

After different tests I got to a "workaround": since the problem seemed to be the cursor, I just moved the creation of the cursor INSIDE THE LOOP.

hc = 0.00833333 # halfcell
insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','cellname'])
i = 1
with arcpy.da.SearchCursor("meteo_xy_map_new_full",['SHAPE@','celly','cellx'] ) as readCursor:
    if ((i%3000)==0):
        del insertCursor
        insertCursor = arcpy.da.InsertCursor('meteogrid_polygon', ['SHAPE@','cellname'])
        ...

This way the cursor gets recreated each 3000 inserts and stops reaching that mysterious limit causing ArcGIS to crash.

What I could see is that legacy ArcObject insert cursors had a flushing mechanism that got buried by the python wrappers... ..but its implemetation has some flaw or assumes some unclarified usage practice.

Improve this answer
2
  • I had a project where I had to test the performance of loading 200 million features across several tables, one of which was 68m rows. I had no problems with 10.3.1, 10.4.0, or 10.4.1. I recently loaded 20m features with a DA InsertCursor using 10.4.1, 10.5.0, 10.5.1 and 10.6.0, with no difficulty. I'm not convinced there were any changes since DA cursors were released at 10.1.
    – Vince
    Commented Mar 17, 2018 at 16:16
  • You can accept your own answer with the checkbox
    – Bera
    Commented Aug 31, 2018 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.