I'm trying to run a python script that processes 2 layers containing roughly 6000 and 15000 polygons each of various shapes and sizes in a loop and outputs a line layer. It runs each polygon through about 5 qgis processes as well as other python operations like matching up id's between features in the two layers and filtering. My guess is that the program is at most O(n^2) asymptotically. While I realize that this isn't exactly optimal, I feel like it should be able to run the program without crashing. It almost always freezes and then crashes and the only way to get it to run all the way through is to split the input up into sections and running it bit by bit.
I have found while googling that heavy tasks tend to make the QGIS gui not respond while running but I'm trying to find some way to make it not crash even if the runtime is rather long (at most 10 minutes).
I tried converting the program to a QGIS model and running it through there, which worked, but it made the program run about 6x slower. The same data which the model processes in 1h10m ran in 9 minutes through the regular python code editor.
I've also tried running the program as a background task using QgsTasks and the task manager, but from what I could find it didn't really suit the type of task my program is made for. Also, I'm not using any print statements in the loop since I know that can cause issues.
Are there any known ways to make Python code in QGIS not crash when running a program like this? Or is my only option to use the slower model if I want the program to be stable?
For reference, the code which I believe causes the most problem is the following. This all runs in a loop for all 6000 polygons and splits the polygon if a line it creates doesn't intersect with a buffer inside the polygon.
result_layer = None
for point in points:
for edge in edges:
if not point.geometry().intersects(buffer.geometry()):
# draw line to the closest point on the longest edge of the current polygon
geom = edge.geometry().closestSegmentWithContext(point.geometry().asPoint())
closest_line = QgsLineString([point.geometry().asPoint(), geom[1]])
closest_line.extend(150,150)
# perform split if line doesn't intersect with buffer
if not QgsGeometry.fromPolyline(closest_line).intersects(buffer.geometry()):
line_layer = QgsVectorLayer('LineString?crs=epsg:3006', 'SPLIT_LINE', 'memory')
prov00 = line_layer.dataProvider()
line_layer.startEditing()
feat = QgsFeature()
feat.setGeometry(QgsGeometry.fromPolyline(closest_line))
prov00.addFeature(feat)
line_layer.updateExtents()
line_layer.commitChanges()
result_layer = processing.run("native:splitwithlines",
{
'INPUT': POLYGON_LAYER,
'LINES': line_layer,
'OUTPUT': "TEMPORARY_OUTPUT"
}
)['OUTPUT']
break
if result_layer is not None:
clip = processing.run("native:clip",
{
'INPUT': line_layer,
'OVERLAY': POLYGON_LAYER,
'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
}
)["OUTPUT"]
TL;DR: Python program freezes and crashes QGIS when running on large enough dataset. Works when splitting input into smaller pieces and running them through program one at a time.
