PyQGIS 3: Cannot Pass Memory OUTPUT To Next Processing Algorithm As INPUT

Question

I am attempting to convert some ArcGIS arcpy command-line scripts to PyQGIS and am running into a problem working with memory layers. The script takes several vector files as input and progresses through a series of transformations to generate a final output. The original script performs those transformations in memory, rather than on disk, to improve overall performance. I have attempted to do the same in QGIS but have been unable to pass an output memory layer to the next process in line as its input. The only thing that I have been able to get to work is to first save the output to file, then use that file's path as input. I have tried various things like passing the output layer's id, name or source, but nothing works.

I've created a small example in which I read in and merge 3 Shapefiles, then attempt to pass the output layer to the dissolve algorithm. The exception that gets thrown is shown at the end. Note that if I pass the layer source with a prefix of "memory://", no error is thrown, but no output is generated. I am submitting the code through the QGIS Python console (3.16.11-Hannover).

import qgis.core
import os


def main():
    sourceFiles = [r'D:\Path\To\Some\Shapefiles\\Source_File_1.shp',
                   r'D:\Path\To\Some\Shapefiles\\Source_File_2.shp',
                   r'D:\Path\To\Some\Shapefiles\\Source_File_3.shp']
    tempDir     = r'D:\TEMP"

    # Merge Input Files
    procParams  = \
    {
        'LAYERS' : sourceFiles,
        'CRS'    : QgsCoordinateReferenceSystem('EPSG:3857'),
        'OUTPUT' : 'TEMPORARY_OUTPUT'
    }
    result       = processing.run("native:mergevectorlayers", procParams)
    mergedLayers = result['OUTPUT']
    mergedName   = "Merge_3_Files"
    mergedPath   = os.path.join(tempDir, mergedName + ".shp")
    msgText      = "Save To Merged File To: %s" % (mergedPath)
    print(msgText)
    QgsVectorFileWriter.writeAsVectorFormat(
                mergedLayers, mergedPath, 'utf-8', driverName='ESRI Shapefile')

    # Dissolve Merged Files - Tried Using The Following INPUTs, Only "mergedPath" Works
    dissolvedURI  = mergedLayers.id()
    ##dissolvedURI  = mergedLayers.name()
    ##dissolvedURI  = mergedLayers.source()
    ##dissolvedURI  = "memory://" + mergedLayers.source()
    ##dissolvedURI  = mergedPath
    procParams    = \
    {   'INPUT' :QgsProcessingFeatureSourceDefinition(
                 dissolvedURI,
                 selectedFeaturesOnly=False,
                 featureLimit=-1,
                 flags=QgsProcessingFeatureSourceDefinition.FlagOverrideDefaultGeometryCheck,
                 geometryCheck=QgsFeatureRequest.GeometryNoCheck),
        'FIELD' :[],
        'OUTPUT':'TEMPORARY_OUTPUT'
    }
    result        = processing.run("native:dissolve", procParams)
    dissolvedLayer= result['OUTPUT']
    dissolvedName = "Dissolve_3_Files"
    dissolvedPath = os.path.join(tempDir, dissolveName + ".shp")
    msgText       = "Save To Dissolved File To: %s" % (dissolvedPath)
    print(msgText)
    QgsVectorFileWriter.writeAsVectorFormat(
            dissolvedLayer, dissolvedPath, 'utf-8', driverName='ESRI Shapefile')

################################################################################
main()



##RESULT
##------
## exec(open('C:/Temp/tmp_othe4_4.py'.encode('utf-8')).read())
## Save To Merged File To: D:\TEMP\Merge_3_Files.shp
##Traceback (most recent call last):
##  File "C:\PROGRA~1\QGIS\apps\Python39\lib\code.py", line 90, in runcode
##    exec(code, self.locals)
##  File "<input>", line 1, in <module>
##  File "<string>", line 53, in <module>
##  File "<string>", line 43, in main
##  File "C:\PROGRA~1/QGIS/apps/qgis-ltr/./python/plugins\processing\tools\general.py", line 108, in run
##    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
##  File "C:\PROGRA~1/QGIS/apps/qgis-ltr/./python/plugins\processing\core\Processing.py", line 168, in runAlgorithm
##    raise QgsProcessingException(msg)
##_core.QgsProcessingException: Unable to execute algorithm
##Could not load source layer for INPUT: Merged_724ac35a_0ace_41d5_ad52_f174adfa02cb not found

Answer 1

Because you are using a QgsProcessingFeatureSourceDefinition class object, you need to add the intermediate temporary output layer to the project, then pass its source string to the class constructor. You can pass an argument to the addMapLayer() method so that the intermediate layer will not be added to the legend.

Also, you don't really need to use QgsVectorFileWriter, if you want a permanent file as your final processing output, just set a valid file path (including extension such as .shp or .gpkg) value to the 'OUTPUT' parameters key.

Try the script below (tested in 3.16):

def main():
    
    sourceFiles = ['D:\\Path\\To\\Some\\Shapefiles\\Source_File_1.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_2.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_3.shp']

    result_1 = processing.run("native:mergevectorlayers",
                    {'LAYERS':sourceFiles,
                    'CRS':QgsCoordinateReferenceSystem('EPSG:3857'),
                    'OUTPUT':'TEMPORARY_OUTPUT'})

    #Add temporary output layer to project to pass its source string to QgsProcessingFeatureSourceDefinition class constructor
    #setting the 2nd argument to False prevents the layer being added to the legend
    QgsProject.instance().addMapLayer(result_1['OUTPUT'], False)

    processing.runAndLoadResults("native:dissolve",
                    {'INPUT':QgsProcessingFeatureSourceDefinition(
                    result_1['OUTPUT'].source(),
                    selectedFeaturesOnly=False,
                    featureLimit=-1,
                    flags=QgsProcessingFeatureSourceDefinition.FlagOverrideDefaultGeometryCheck,
                    geometryCheck=QgsFeatureRequest.GeometryNoCheck),
                    'FIELD':[],
                    'OUTPUT':'TEMPORARY_OUTPUT'
                    #To create a file output pass a path and extention to output param (like below)
                    #'OUTPUT':'D:\\Path\\To\\Directory\\Dissolve_3_Files.shp'
                    })
                    
main()

On the other hand, if you don't particularly need to use QgsProcessingFeatureSourceDefinition, then you can simply pass results['OUTPUT'] directly from the first algorithm as the input to the second algorithm. E.g.

def main():

    sourceFiles = ['D:\\Path\\To\\Some\\Shapefiles\\Source_File_1.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_2.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_3.shp']

    result_1 = processing.run("native:mergevectorlayers",
                    {'LAYERS':sourceFiles,
                    'CRS':QgsCoordinateReferenceSystem('EPSG:3857'),
                    'OUTPUT':'TEMPORARY_OUTPUT'})
                    

    processing.runAndLoadResults("native:dissolve",
                    {'INPUT':result_1['OUTPUT'],
                    'FIELD':[],
                    'OUTPUT':'TEMPORARY_OUTPUT'
                    #To create a file output pass a path and extention to output param (like below
                    #'OUTPUT':'D:\\Path\\To\\Directory\\Dissolve_3_Files.shp'
                    })
                    
main()

Based on further clarification in comments, I tested running this as standalone script from the command line (with pyqgis environment properly set) and it worked fine for me using the script below (with my own paths of course).

# See https://gis.stackexchange.com/questions/279874/using-qgis-3-processing-algorithms-from-pyqgis-standalone-scripts-outside-of-gu
import sys

import os

from qgis.core import (
     QgsApplication, 
     QgsCoordinateReferenceSystem,
     QgsProject,
     QgsProcessingFeatureSourceDefinition,
     QgsFeatureRequest
)
from qgis.analysis import QgsNativeAlgorithms

# See https://gis.stackexchange.com/a/155852/4972 for details about the prefix 
QgsApplication.setPrefixPath('C:/OSGeo4W/apps/qgis', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Append the path where processing plugin can be found
sys.path.append('C:\\OSGeo4W\\apps\\qgis\\python\\plugins')

import processing
from processing.core.Processing import Processing
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

sourceFiles = ['D:\\Path\\To\\Some\\Shapefiles\\Source_File_1.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_2.shp',
                   'D:\\Path\\To\\Some\\Shapefiles\\Source_File_3.shp']

result_1 = processing.run("native:mergevectorlayers",
                {'LAYERS':sourceFiles,
                'CRS':QgsCoordinateReferenceSystem('EPSG:3857'),
                'OUTPUT':'TEMPORARY_OUTPUT'})
                
#Add temporary output layer to project to pass its source string to QgsProcessingFeatureSourceDefinition class constructor
QgsProject.instance().addMapLayer(result_1['OUTPUT'], False)

processing.run("native:dissolve",
                {'INPUT':QgsProcessingFeatureSourceDefinition(
                result_1['OUTPUT'].source(),
                selectedFeaturesOnly=False,
                featureLimit=-1,
                flags=QgsProcessingFeatureSourceDefinition.FlagOverrideDefaultGeometryCheck,
                geometryCheck=QgsFeatureRequest.GeometryNoCheck),
                'FIELD':[],
                #To create a file output pass a path and extention to output param (like below
                'OUTPUT':'D:\\Path\\To\\Directory\\Dissolve_3_Files.shp'
                })

Links to a few other relevant questions and answers for reference:

Running PyQGIS algorithm on selected features in layer

Using QGIS 3 Processing algorithms from PyQGIS standalone scripts (outside of GUI)

Layer is not valid error in my standalone PyQGIS script/app

Answer 2

UPDATE:
The Help instructions given by processing.algorithmHelp("native:dissolve") suggest that the following INPUTS are acceptable:

• str: layer ID
• str: layer name
• str: layer source
• QgsProcessingFeatureSourceDefinition
• QgsProperty
• QgsVectorLayer

Yet when I tried to pass a QgisVectorLayer directly, I encountered this error:

results, ok = alg.run(parameters, context, feedback, {}, False) _core.QgsProcessingException: Feature (99) from “Merged” has invalid geometry. Please fix the geometry or change the Processing setting to the “Ignore invalid input features” option.

It was for this reason that I used:
QgsProcessingFeatureSourceDefinition(,
dissolvedURI,
selectedFeaturesOnly=False,
featureLimit=-1,
flags=QgsProcessingFeatureSourceDefinition.FlagOverrideDefaultGeometryCheck,
geometryCheck=QgsFeatureRequest.GeometryNoCheck),

which in turn required the QgsProject.instance().addMapLayer(mergedLayers, False) call.

A colleague, however, suggested that I first pass the incoming Shapefiles through "native:fixgeometries". Many of our files are derived from raw satellite images interpreted and converted from rasters to polygon vectors in an earlier step. It's been observed that this conversion may introduce errors that can be cleaned up with a fixgeometries step. Apparently that's what the above QgsProcessingFeatureSourceDefinition call has also been doing for me. When I use fixgeometries, I find that I now no longer need the addMapLayer call – I can pass the memory layer directly. This works:

import qgis.core
import os


def main():
    sourceFiles  = [r'D:\Path\To\Some\Shapefiles\\Source_File_1.shp',
                    r'D:\Path\To\Some\Shapefiles\\Source_File_2.shp',
                    r'D:\Path\To\Some\Shapefiles\\Source_File_3.shp']
    tempDir      = r'D:\TEMP"

    # Merge Input Files
    procParams   = \
    {
        'LAYERS' : sourceFiles,
        'CRS'    : QgsCoordinateReferenceSystem('EPSG:3857'),
        'OUTPUT' : 'TEMPORARY_OUTPUT'
    }
    result       = processing.run("native:mergevectorlayers", procParams)
    mergedLayers = result['OUTPUT']

    # Fix Merged Layer
    procParams   = \
    {   'INPUT' :mergedLayers,
        'OUTPUT':'TEMPORARY_OUTPUT'
    }
    result       = processing.run("native:fixgeometries", procParams)
    fixedLayers  = result['OUTPUT']

    # Dissolve Fixed Layers
    procParams    = \
    {   'INPUT' :fixedLayers,
        'FIELD' :[],
        'OUTPUT':'TEMPORARY_OUTPUT'
    }
    result        = processing.run("native:dissolve", procParams)
    dissolvedLayer= result['OUTPUT']

    # Continue to next step... (or dump to file as is being done here)
    dissolvedName = "Dissolve_3_Files"
    dissolvedPath = os.path.join(tempDir, dissolvedName + ".shp")
    msgText       = "Save To Dissolved File To: %s" % (dissolvedPath)
    print(msgText)
    QgsVectorFileWriter.writeAsVectorFormat(
            dissolvedLayer, dissolvedPath, 'utf-8', driverName='ESRI Shapefile')

################################################################################
main()