Memory leak in ArcGIS Pro: in_memory workspace is not deleted

Question

I have a python script for ArcGIS Pro 2.0.1 that executes the same code with the same data size in a loop. So intuitively the run times of the iterations of the loop shall be about the same. However, each iteration takes more time than the previous (with the growing problem size the times start to double). The only thing that I have found that could explain it is that the memory consumption is always growing (which leads to crushing with the larger datasets). I have worked with a memory profiler and the problem might be with the Delete_management and DeleteFeature_management not releasing the memory.

I have also printed the feature classes contained in the in_memory workspace, and the "deleted" feature classes are not appearing there, but the memory is not released.

How can I manage memory use in arcpy with arcgis pro?

So far all the advise were not to forget to delete in_memory workspace, but this is exactly what does not work for some reason.

Could it be due to the multi-processing in arcgis pro?

The problem happens with this function of mine, the problem starts appearing when I save individually generated lines (points to line) to a temporary file lines_out.

def connect_gabriel_graph(points, diameter_dict, n_inter, n_runs, output_dir):
'''
This function connects the two points (i,j) iff within the d_ij there is no point k, such that 
    d_ij^2 >= d_ik^2 + d_jk^2, or there is no point k in the radius of d_ij


:param points: points feature class path
:param diameter_dict: all the distances combinations, i.e., {1:{2: 15, 3: 20}, 2: {3: 10}}
:param n_inter: number of points
:param n_runs: run id (number)
:return: path to the created lines
'''

# Create a point layer
points_layer_name = os.path.join('in_memory', 'points_tmp_int{0}_run{1}'.format(n_inter, n_runs))
points_layer = arcpy.MakeFeatureLayer_management(points, points_layer_name)

# Get the spatial reference (as in the points)
spatial_ref_in = arcpy.Describe(points).spatialReference

# Create a featureclass to store the resulting links
links_out_name = 'links_int{0}_run{1}'.format(n_inter, n_runs)
all_links_in = arcpy.CreateFeatureclass_management('in_memory', links_out_name, 'POLYLINE',
                                                   spatial_reference=spatial_ref_in)

# Create a tmp layer to store the current link (defining the path and name)
lines_name = 'lines_tmp'
lines_out = os.path.join('in_memory', lines_name)

points_field_objects = arcpy.ListFields(points_layer)
points_fields = [field.name for field in points_field_objects if field.type != 'Geometry']


# Iterate through all the points with the IDs in the provided range
for i in range(1, n_inter):
    # Iterate through all other points
    for j in diameter_dict[i]:
        # Get the lengths
        ij_len = diameter_dict[i][j]
        # Iterate though all other points
        for k in range(1, n_inter + 1):
            # Check that these are not the i and j themselves
            if k != i and k != j:
                # Get the distance between i and k as we do not have the permutations, we need to pay attention,
                # where the distance is stored
                if k > i:
                    ik_len = diameter_dict[i][k]
                elif k < i:
                    ik_len = diameter_dict[k][i]

                if k > j:
                    jk_len = diameter_dict[j][k]
                elif k < j:
                    jk_len = diameter_dict[k][j]

                # We define, when to connect (i,j)
                if j != n_inter:
                        # for the normal case the last k will have the id == n_inter_in
                        threshold = n_inter
                else:
                        # if j == n_inter_in then we can have two cases of the last k
                        if i == n_inter - 1:
                            threshold = n_inter - 2
                        else:
                            threshold = n_inter - 1
                # We check that the connection condition always holds. As soon as the condition is violated,
                # the link is dismissed
                if math.pow(ij_len, 2) >= math.pow(ik_len, 2) + math.pow(jk_len, 2):
                    break

                # If we see that it is the last k (the cycle was not brocken before) and the condition holds,
                # there shall be a link
                elif math.pow(ij_len, 2) < math.pow(ik_len, 2) + math.pow(jk_len, 2) and k == threshold:
                    # We select the (i,j) from all the points
                    if "OID" in points_fields:
                        # Define the SQL expression for the points with the IDs with the range
                        clause_in = '"OID" = {0} OR "OID" = {1}'.format(i, j)

                    elif "OBJECTID" in points_fields:
                        # Define the SQL expression for the points with the IDs with the range
                        clause_in = '"OBJECTID" = {0} OR "OBJECTID" = {1}'.format(i, j)

                    arcpy.SelectLayerByAttribute_management(points_layer, 'NEW_SELECTION', clause_in)

                    # Create a line between them in a tmp layer
                    arcpy.PointsToLine_management(points_layer, lines_out)

                    # Add the Origin and Destination fields
                    arcpy.AddField_management(lines_out, 'OriginID', 'LONG')
                    arcpy.AddField_management(lines_out, 'DestinationID', 'LONG')

                    # Populate the fields with the (i,j) IDs
                    with arcpy.da.UpdateCursor(lines_out, ["OriginID", 'DestinationID']) as cursor:
                        for row in cursor:
                            row[0] = i
                            row[1] = j
                            cursor.updateRow(row)

                    arcpy.SelectLayerByAttribute_management(points_layer, 'CLEAR_SELECTION')

                    # Append the link to the created in this process feature class
                    arcpy.Append_management(lines_out, all_links_in, 'NO_TEST')

Answer 1

The problem in this case was not directly the 'in_memory' workspace, but rather a generic memory leak. The memory profiler did not really help as it for some reason does not see, what happens in the 'in_memory' workspace and shows that nothing was deleted, although there are no feature classes.

After manually controlling the size of all the lists/dictionaries in the function, checking that all the feature classes are overwritten or deleted, I have found the memory leak source:

# Create a line between the two selected points in a tmp layer
arcpy.PointsToLine_management(points_layer, lines_out)

This function for some reason does not behave well in a for loop, causing the run time grow from iteration to iteration (sometimes doubling, sometimes increasing by a third) and making it impossible to conduct multiple runs especially with bigger problems.

I have changed the function to:

array_tmp = arcpy.Array()
with arcpy.da.SearchCursor(points_layer, ['SHAPE@X', 'SHAPE@Y']) as cursor:
       for row in cursor:
           point.X = row[0]
           point.Y = row[1]
           array_tmp.append(point)

features_lines.append(
        arcpy.Polyline(array_tmp, spatial_ref_in))

and the run time does not grow like this anymore. This code was a small adaptation of this one. In my case, this implementation is also generally faster than the PointsToLine_management.

Answer 2

If you change

lines_out = os.path.join('in_memory', lines_name)

to

lines_out = os.path.join('%scratchgdb%', lines_name)

does your profile report better memory clearing in the Delete/Delete Features calls? This will use a scratch file geodatabase instead of an in-memory geodatabase.