0

I have the following PyQGIS script that is a combination of personal edits and graphical modeler:

    cities_layer = self.parameterAsVectorLayer(parameters, 'cities', context)
    urban_area = self.parameterAsVectorLayer(parameters, 'urbanarea', context)
    
    # Create a temporary vector layer to store the selected points
    selected_points_layer = QgsVectorLayer("Point?crs=" + cities_layer.crs().authid(), "selected_points", "memory")
    selected_points_layer.startEditing()
    selected_points_layer.dataProvider().addAttributes(cities_layer.fields())
    selected_points_layer.updateFields()
    
    total_features = cities_layer.featureCount()
    current_step = 0
    
    # Create spatial index UrbanArea
    alg_params = {
        'INPUT': parameters['urbanarea']
    }
    outputs['CreateSpatialIndexUrbanarea'] = processing.run('native:createspatialindex', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

    feedback.setCurrentStep(1)
    if feedback.isCanceled():
        return {}

    # Create spatial index Cities
    alg_params = {
        'INPUT': parameters['cities']
    }
    outputs['CreateSpatialIndexCities'] = processing.run('native:createspatialindex', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

    feedback.setCurrentStep(2)
    if feedback.isCanceled():
        return {}

    # Loop through all features in the 'cities' layer by selecting each feature
    for feature_id in cities_layer.allFeatureIds():
        current_step += 1
        feedback.setProgress(current_step)
        cities_layer.select(feature_id)

        # Access the selected feature's attributes
        city_feature = cities_layer.selectedFeatures()[0]
        city_name = city_feature['Name']
        
        # Create a new temporary vector layer to store the selected point
        selected_points_layer = QgsVectorLayer("Point?crs=" + cities_layer.crs().authid(), "selected_points", "memory")
        selected_points_layer.startEditing()
        selected_points_layer.dataProvider().addAttributes(cities_layer.fields())
        selected_points_layer.updateFields()
        
        # Add the selected feature to the temporary layer
        selected_points_layer.addFeature(city_feature)

        # Save the temporary layer to the output
        selected_points_layer.commitChanges()
        
        # Extract by location for the current selected city feature
        alg_params = {
            'INPUT': outputs['CreateSpatialIndexUrbanarea']['OUTPUT'],
            'INTERSECT': selected_points_layer,
            'PREDICATE': [0],  # intersect
            'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
        }
        outputs['ExtractByLocation'] = processing.run('native:extractbylocation', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
        results['ExtractLocation'] = outputs['ExtractByLocation']['OUTPUT']
        
        canvas = iface.mapCanvas()
        
        result_layer = QgsProcessingUtils.mapLayerFromString(outputs['ExtractByLocation']['OUTPUT'], context)
        extent_ua = result_layer.extent()
        canvas.setExtent(extent_ua)
        scale = canvas.scale()
        
        canvas.setExtent(city_feature.geometry().boundingBox())
        canvas.zoomScale(scale*2) 
        
        # Clear the selection and temporary layer after each iteration
        selected_points_layer.startEditing()
        selected_points_layer.selectAll()
        selected_points_layer.deleteSelectedFeatures()
        selected_points_layer.commitChanges()
        
        # refresh the canvas
        canvas.refresh()
        extent = iface.mapCanvas().extent()
        
        # Build query inside an extent - roads
        alg_params = {
            'EXTENT': extent,
            'KEY': 'highway',
            'SERVER': 'https://lz4.overpass-api.de/api/interpreter',
            'TIMEOUT': 25,
            'VALUE': ''
        }
        outputs['BuildQueryInsideAnExtentRoads'] = processing.run('quickosm:buildqueryextent', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
        
        feedback.setCurrentStep(3)
        if feedback.isCanceled():
            return {}
        
        # Build query inside an extent - buildings
        alg_params = {
            'EXTENT': extent,
            'KEY': 'building',
            'SERVER': 'https://lz4.overpass-api.de/api/interpreter',
            'TIMEOUT': 25,
            'VALUE': ''
        }
        outputs['BuildQueryInsideAnExtentBuildings'] = processing.run('quickosm:buildqueryextent', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
        
        feedback.setCurrentStep(4)
        if feedback.isCanceled():
            return {}
    
        # Open sublayers from an OSM file - roads
        alg_params = {
            'FILE': outputs['BuildQueryInsideAnExtentRoads']['OUTPUT_URL'],
            'OSM_CONF': ''
        }
        outputs['OpenSublayersFromAnOsmFileRoads'] = processing.run('quickosm:openosmfile', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

        feedback.setCurrentStep(5)
        if feedback.isCanceled():
            return {}

        # Rename layer - Roads
        alg_params = {
            'INPUT': outputs['OpenSublayersFromAnOsmFileRoads']['OUTPUT_LINES'],
            'NAME': 'Roads_OSM'
        }
        outputs['RenameLayerRoads'] = processing.run('native:renamelayer', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
        results['Gis_osm_roads_free_1'] = outputs['RenameLayerRoads']['OUTPUT']
        
        feedback.setCurrentStep(6)
        if feedback.isCanceled():
            return {}

        # Open sublayers from an OSM file - buildings
        alg_params = {
            'FILE': outputs['BuildQueryInsideAnExtentBuildings']['OUTPUT_URL'],
            'OSM_CONF': ''
        }
        outputs['OpenSublayersFromAnOsmFileBuildings'] = processing.run('quickosm:openosmfile', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

        feedback.setCurrentStep(7)
        if feedback.isCanceled():
            return {}

        # Rename layer - Buildings
        alg_params = {
            'INPUT': outputs['OpenSublayersFromAnOsmFileBuildings']['OUTPUT_MULTIPOLYGONS'],
            'NAME': 'Buildings_OSM'
        }
        outputs['RenameLayerBuildings'] = processing.run('native:renamelayer', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
        results['Gis_osm_buildings_a_free_1'] = outputs['RenameLayerBuildings']['OUTPUT']
        
        feedback.setCurrentStep(8)
        if feedback.isCanceled():
            return {}

        # Set layer style (gis_osm_roads_free_1)
        alg_params = {
            'INPUT': outputs['RenameLayerRoads']['OUTPUT'],
            'STYLE': 'D:\\Layer_styles\\one_road_style.qml'
        }
        outputs['SetLayerStyleGis_osm_roads_free_1'] = processing.run('native:setlayerstyle', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

        feedback.setCurrentStep(9)
        if feedback.isCanceled():
            return {}
        
        # Set layer style (gis_osm_buildings_a_free_1)
        alg_params = {
            'INPUT': outputs['RenameLayerBuildings']['OUTPUT'],
            'STYLE': 'D:\\Layer_styles\\osm_buildings_style.qml'
        }
        outputs['SetLayerStyleGis_osm_buildings_a_free_1'] = processing.run('native:setlayerstyle', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

        feedback.setCurrentStep(10)
        if feedback.isCanceled():
            return {}

        project = QgsProject().instance()
        layout = project.layoutManager().layoutByName('Test')
        
        scale = canvas.scale()
        canvas.zoomScale(scale-(scale*0.2))
        canvas.refresh()
        extent2 = canvas.extent()
        
        map_item = layout.itemById('Map 1')
        map_item.zoomToExtent(extent2)
        map_item.refresh()
        
        gis_osm_roads_free_1_lyr = context.getMapLayer(results['Gis_osm_roads_free_1'])
        gis_osm_buildings_a_free_1_lyr = context.getMapLayer(results['Gis_osm_buildings_a_free_1'])
        
        layers_to_add = [
            gis_osm_roads_free_1_lyr,
            gis_osm_buildings_a_free_1_lyr
        ]
        map_item.setLayers(layers_to_add)
        
        scalebar_item = layout.itemById("Scalebar")
        scalebar_item.setLinkedMap(map_item)
        scalebar_item.applyDefaultSize()
        title = layout.itemById("Title")
        Saving_Name = city_name
        
        title.setText(Saving_Name)
        layout.refresh()
        
        directory = r"D:\Data\Exports"
        jpeg_path = os.path.join(directory, Saving_Name + '.jpeg')
        
        exporter = QgsLayoutExporter(layout)
        settings = QgsLayoutExporter.ImageExportSettings()
        settings.dpi = 300
    
        exporter.exportToImage(jpeg_path, settings)

"cities_layer" is is a point data representing multiple cities in the world. "urban_area" represents a polygon that represents urban extent.

Basically, what I am trying into a for statement is:

  • Select each city point zoom to it's location. To keep track of selected features those are added to a temporary point layer.
  • Extract by location urban_area polygon and set canvas extent to it's extent.
  • Zoom out a bit and with QuickOSM download based on canvas extent, the roads and buildings.
  • Rename algorithm was used because otherwise I could have not open sublayers to set their style.
  • After downloading data and setting the style, those temporary output were added to the map item "Map 1" in order to be displayed. Layers not being added to the Project.
  • Then I export the map, clear selection and this process tries to repeat this for each feature in cities_layer.

Issue:

After 20 iterations say for small cities or 10 for bigger cities, the QGIS blocks in various locations of the process and it will not continue the iteration.

Assuming that the blocking might be due to bad memory handling:

  • Since there is no temporary layer loaded to the QGIS project, the shapefiles from processing algorithms should be automatically deleted by QGIS.

  • The error might be that I load those results to the layout exporter/map item.

  • I have tried redefining the map item layers with a null list but with no results:

            map_item.setLayers([])
            map_item.refresh()

Question:

Not knowing if the data from map item remains after each iteration, how do I make sure that I do not overload it?

Additional information based on suggestions in comments:

I have tested by commenting out line by line. I have left running only the "build raw query" and "open sublayers" processing algorithms. The issue still persists and it blocks at opening sublayers.

  1. This is where the log/process stopped:

I have made a feedback.pushinfo() before each process to see exactely where the log stops. enter image description here

  1. It also appeared an error :

enter image description here

The instruction at 0x00007FFE5EBE7DAE referenced memory at 0x0000000000000004. The memory could not be read.

"Error message commonly seen in Windows operating systems. It typically indicates a problem with memory access by a program or process. This error message can be caused by various factors, such as software bugs, corrupted system files, faulty hardware, or issues with system memory (RAM)."

From here in task manager RAM was used on QGIS and other windows processes but script did not wok, there was no disk or network usage that can indicate download of OSM data.

7
  • 1
    Do you get any error msg? It just stops without any error msg?
    – Nil
    Commented Mar 1 at 9:17
  • No error message. It blocks on the loading bar at certain percent and it remains there. The disk usage, CPU and internet go down to 0, so there is no activity. If I try to interact with it or to cancel the process it will force the project into "Not responding". Commented Mar 1 at 20:45
  • 2
    There is a huge amount of code here which makes it very hard to debug. Have you tried working backwards and commenting out a process at a time to see if you can isolate the problem?
    – Matt
    Commented Mar 5 at 9:21
  • I have tried to put feedback.pushinfo() to each line/process to see into the log file where it stops. Now it stops at 26%-36%. That is right after the the download (build raw query), into opening sublayers processing algorithm. Commented Mar 5 at 13:20
  • Previously it blocked also further on, at 89%, being blocked at an extract by attribute. I had before some extract by attributes of some road categories where I styled them separately, but now I have only one style for the roads. This is why my guesses were that is something with the memory because it does not stop at the quick OSM query so I can say that API blocks at certain amount of requests. To eliminate this possibility have also set a timer for each iteration, and the issue still persists. Commented Mar 5 at 13:21

1 Answer 1

0

I have found out the issue that blocked the for statement. It was on the code snippets generated by graphical modeler.

More specifically on each processing.run() there is a is_child_algorithm=True.

This statement being "True", will transform the processing.run() output as a string reference, instead of the actual layer object.

This is used for complex processes so when you use this output for another processing.run() as input it won't create a copy, instead it will use the original output that is somehow "stored" and that string has the path to that temporary output.

So if you have any issue with a "for" statement that runs some graphical modeler "processing.run()" or you want to interact with the vector layer within it you can delete "is_child_algorithm=True" so you wont need to use methods such as QgsProcessingUtils.mapLayerFromString().

Possible issue: I guess the reference for previous iterations remained stored and after the cache is full it won't perform next iteration.

If you use the output just for another processing.run() as input it is fine but with any other interaction you need to use some methods to get to those geometries.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.