Using QGIS Processing algorithms from PyQGIS standalone scripts (outside of GUI)

Question

I am writing a script which must work outside of QGIS GUI. I call some API functions from qgis.core but I would like to use the processing plugin.

I am able to import processing with sys.path.append() but I cannot run any process. Moreover, all "native" algs are missing in QgsApplication.processingRegistry().algorithms()

Is it possible to run processing that way?

import os, sys
from qgis.core import *

QgsApplication.setPrefixPath('/usr', True)
qgs = QgsApplication([], False)
qgs.initQgis()

sys.path.append('/usr/share/qgis/python/plugins')
from processing.core.Processing import Processing
Processing.initialize()
import processing

layer1 = QgsVectorLayer('data/ROUTE_PRIMAIRE.SHP')
layer2 = QgsVectorLayer('data/ROUTE_SECONDAIRE.SHP')

processing.run('qgis:union', layer1, layer2, 'test.shp') # returns nothing

I am using QGIS 3.0.1 on Debian 9.

Answer 1

You can run a QGIS Processing algorithm in standalone (no GUI) mode in this way:

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer
)

# See https://gis.stackexchange.com/a/155852/4972 for details about the prefix 
QgsApplication.setPrefixPath('/usr', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Append the path where processing plugin can be found
sys.path.append('/docs/dev/qgis/build/output/python/plugins')

import processing
from processing.core.Processing import Processing
Processing.initialize()

layer1 = QgsVectorLayer('/path/to/geodata/lines_1.shp', 'layer 1', 'ogr')
layer2 = QgsVectorLayer('/path/to/geodata/lines_2.shp', 'layer 2', 'ogr')

# You can see what parameters are needed by the algorithm  
# using: processing.algorithmHelp("qgis:union")
params = { 
    'INPUT' : layer1,
    'OVERLAY' : layer2, 
    'OUTPUT' : '/path/to/output_layer.gpkg|layername=output'
}
feedback = QgsProcessingFeedback()

# See https://gis.stackexchange.com/a/276979/4972 for a list of algorithms
res = processing.run('qgis:union', params, feedback=feedback)
res['OUTPUT'] # Access your output layer

Note for QGIS versions LESS THAN 3.16.5

If you want to use a native algorithm (i.e., an algorithm from the native provider, whose algorithms are written in C++), you need to add the provider after initializing Processing:

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer
)
from qgis.analysis import QgsNativeAlgorithms

# See https://gis.stackexchange.com/a/155852/4972 for details about the prefix 
QgsApplication.setPrefixPath('/usr', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Append the path where processing plugin can be found
sys.path.append('/docs/dev/qgis/build/output/python/plugins')

import processing
from processing.core.Processing import Processing
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

layer = QgsVectorLayer('/path/to/geodata/lines.shp', 'my layer', 'ogr')

# You can see what parameters are needed by the algorithm  
# using: processing.algorithmHelp("native:extractvertices")
params = {
    'INPUT': layer,
    'OUTPUT': 'memory:'
}
feedback = QgsProcessingFeedback()

# See https://gis.stackexchange.com/a/276979/4972 for a list of algorithms
res = processing.run("native:extractvertices", params, feedback=feedback)
res['OUTPUT'] # Access your output layer

---

Note: If you actually wrote your own Processing algorithm and would like to run it in a standalone PyQGIS script, see: Using custom Processing algorithm from standalone PyQGIS scripts (outside of GUI)

Answer 2

I had an error:

NameError: name 'QgsNativeAlgorithms' is not defined

when I tried to use a native algorithm as above, in an OSGeo4W install that included QGIS 3.4.4.

The solution turned out to be a missing import:

from qgis.analysis import QgsNativeAlgorithms

(from QGIS 3.4/3.6 standalone script)

Answer 3

For standalone QGIS processing, you can now use qgis_process command line (available since QGIS 3.14, doc at https://docs.qgis.org/3.16/en/docs/user_manual/processing/standalone). Because of this, it's now possible to the following

qgis_process run native:union --INPUT=/path/to/geodata/lines_1.shp --OVERLAY=/path/to/geodata/lines_2.shp --OUTPUT="/tmp/output_layer.gpkg

You will suffer from one limitation at the moment compared to Python standalone approach: you can't specify the layer name output e.g https://github.com/qgis/QGIS/issues/38272