I'm trying to run a python script that processes 2 layers containing roughly 6000 and 15000 polygons each of various shapes and sizes in a loop and outputs a line layer. It runs each polygon through about 5 qgis processes as well as other python operations like matching up id's between features in the two layers and filtering. My guess is that the program is at most O(n^2) asymptotically. While I realize that this isn't exactly optimal, I feel like it should be able to run the program without crashing. It almost always freezes and then crashes and the only way to get it to run all the way through is to split the input up into sections and running it bit by bit.

I have found while googling that heavy tasks tend to make the QGIS gui not respond while running but I'm trying to find some way to make it not crash even if the runtime is rather long (at most 10 minutes).

I tried converting the program to a QGIS model and running it through there, which worked, but it made the program run about 6x slower. The same data which the model processes in 1h10m ran in 9 minutes through the regular python code editor.

I've also tried running the program as a background task using QgsTasks and the task manager, but from what I could find it didn't really suit the type of task my program is made for. Also, I'm not using any print statements in the loop since I know that can cause issues.

Are there any known ways to make Python code in QGIS not crash when running a program like this? Or is my only option to use the slower model if I want the program to be stable?

For reference, the code which I believe causes the most problem is the following. This all runs in a loop for all 6000 polygons and splits the polygon if a line it creates doesn't intersect with a buffer inside the polygon.

result_layer = None
for point in points:
    for edge in edges:
        if not point.geometry().intersects(buffer.geometry()):
            # draw line to the closest point on the longest edge of the current polygon
            geom = edge.geometry().closestSegmentWithContext(point.geometry().asPoint())
            closest_line = QgsLineString([point.geometry().asPoint(), geom[1]])
            # perform split if line doesn't intersect with buffer
            if not QgsGeometry.fromPolyline(closest_line).intersects(buffer.geometry()):
                line_layer = QgsVectorLayer('LineString?crs=epsg:3006', 'SPLIT_LINE', 'memory')
                prov00 = line_layer.dataProvider()
                feat = QgsFeature()
                result_layer = processing.run("native:splitwithlines",
                    'INPUT': POLYGON_LAYER,
                    'LINES': line_layer,
                    'OUTPUT': "TEMPORARY_OUTPUT"
    if result_layer is not None:
         clip = processing.run("native:clip", 
              'INPUT': line_layer,
              'OVERLAY': POLYGON_LAYER,
              'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT

TL;DR: Python program freezes and crashes QGIS when running on large enough dataset. Works when splitting input into smaller pieces and running them through program one at a time.

  • Why do you think QgsTasks doesnt really suit the type of task my program is made for? That would be my first attempt.
    – MrXsquared
    Feb 24, 2022 at 14:27
  • @MrXsquared I made an attempt at converting my code to use the task stucture but couldn't find any examples of how to handle the output layer. Most examples and the documentation specifically regarding tasks from functions are about doing numerical operations. Since you can't interact with the main thread and QgsProject for example, I figured it wasn't made for that. When I tried, addMapLayer(result['OUTPUT']) in on_finished crashes QGIS instantly.
    – Axekan
    Feb 24, 2022 at 14:42
  • I see the difficulties, but that is not a reason to not use tasks. Cant rewrite your script now, but maybe this answer can help you by giving an example gis.stackexchange.com/a/405073/107424. Alternatively you could give a custom processing tool a try, if that is easier to implement for you.
    – MrXsquared
    Feb 24, 2022 at 14:52
  • @MrXsquared I understand, I'll have to do some more rigorous attempts at rewriting it. Otherwise I might return to the processing tool route, as I actually made one for use in the graphical modeler previously. Thanks for the tips
    – Axekan
    Feb 24, 2022 at 15:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.