7

I am trying to create a standalone PyQGIS script which runs the Distance Matrix in QGIS 3, see below:


######### INITIALISE QGIS STANDALONE ################

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer,
     QgsField,
     QgsFields,
     QgsProperty,
     QgsProcessingFeatureSourceDefinition,
     QgsProcessingOutputLayerDefinition
)

#start QGIS instance without GUI
QgsApplication.setPrefixPath('/usr/share/qgis', True)
myqgs = QgsApplication([], False)
myqgs.initQgis()

######### INITIALISE THE PROCESSING FRAMEWORK ################

# Append the path where processing plugin can be found (assumes Debian)
sys.path.append('/usr/share/qgis/python/plugins')

#import modules needed
import processing
from processing.core.Processing import Processing

#start the processing module
processing.core.Processing.Processing.initialize()

######### Set Distance Matrix plugin parameters ############################

# I used this command in the QGIS3 python console
# >>> processing.algorithmHelp("qgis:distancematrix") 
# to get the object types it accepts for each parameter

inputlayer = QgsVectorLayer('/path/to/myinputfile.shp', 'layer 1', 'ogr')
targetlayer = QgsVectorLayer('/path/to/mytargetfile.shp', 'layer 2', 'ogr')

inputfield = QgsProperty.fromField("InputPointID")
targetfield = QgsProperty.fromField("TargetPointID")

matrixtype = QgsProperty.fromValue(0)
nearestpoints = QgsProperty.fromValue(0)

outputlayer = QgsProcessingOutputLayerDefinition('/path/to/myoutputfile.csv')

params = { 
    'INPUT' : inputlayer,
    'INPUT_FIELD' : inputfield,
    'TARGET' : targetlayer,
    'TARGET_FIELD' : targetfield,
    'MATRIX_TYPE' : matrixtype,
    'NEAREST_POINTS' : nearestpoints,
    'OUTPUT' : outputlayer
}

feedback = QgsProcessingFeedback()

res = processing.run('qgis:distancematrix', params, feedback=feedback)
res['OUTPUT'] # Access your output layer

The script runs without errors but I don't get an output file. I have looked at the PYQGIS 3 API reference and searched more generally but there are so few worked examples for standalone PYQGIS3.

I have also run the command below in QGIS3 and used the field types it recommends but it still doesn't work:

processing.algorithmHelp("qgis:distancematrix")

Any ideas what could be wrong with my script?

From Failed to create memory layers in QGIS application on Linux, I added the following code to check if the input layers were loading properly.

#Input Layers Validity Check
print(inputlayer.isvalid())
print(targetlayer.isvalid())

I got a False result on both layers.

Then using PyQGIS QgsVectorLayer() Loading Invalid Layer in Standalone Python Script? I found that the input layers were not loading because I had the wrong prefix. I replaced the below.

QgsApplication.setPrefixPath('/usr/share/qgis', True)

With this

QgsApplication.setPrefixPath('/usr', True)

It is now creating a blank csv file as the output (which is progress) but its still throwing an error when it runs, see below. Its a step forward from where it was but still not fixed yet.

Traceback (most recent call last):
  File "/path/to/distmatrixqgis3standalonenogui.py", line 99, in <module>
    res = processing.run('qgis:distancematrix', params, feedback=feedback)
  File "/usr/share/qgis/python/plugins/processing/tools/general.py", line 96, in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
  File "/usr/share/qgis/python/plugins/processing/core/Processing.py", line 183, in runAlgorithm
    raise QgsProcessingException(msg)
_core.QgsProcessingException: There were errors executing the algorithm.
Improve this question
0

2 Answers 2

Reset to default
6

I have found and fixed the final errors. I tried changing how each algorthim parameter got its value.

By changing the code below:


inputfield = QgsProperty.fromField("InputPointID")
targetfield = QgsProperty.fromField("TargetPointID")

To this:


inputfield = QgsProperty.fromValue("InputPointID")
targetfield = QgsProperty.fromValue("TargetPointID")

It has fixed the final problems and the script runs well.

My final working example of the PYQGIS 3 standalone code (Including all updates described above) to run the QGIS 3 Distance Matrix in the OS shell is as below. For info/interest, I will be tweaking it next to run in GNU Parallel; I can do that bit so I won't take up any more time on this question. Thanks to everyone. Question closed.



######### INITIALISE QGIS STANDALONE ################

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer,
     QgsField,
     QgsFields,
     QgsProperty,
     QgsProcessingFeatureSourceDefinition,
     QgsProcessingOutputLayerDefinition
)

#start QGIS instance without GUI
#Make sure the prefix is correct. Even though QGIS is in '/usr/share/qgis',
#the prefix needs to be '/usr' (assumes Debian OS)

QgsApplication.setPrefixPath('/usr', True)
myqgs = QgsApplication([], False)
myqgs.initQgis()

######### INITIALISE THE PROCESSING FRAMEWORK ################

# Append the path where processing plugin can be found (assumes Debian OS)
sys.path.append('/usr/share/qgis/python/plugins')

#import modules needed
import processing
from processing.core.Processing import Processing

#start the processing module
processing.core.Processing.Processing.initialize()

######### Set Distance Matrix plugin parameters ############################

# I used this command in the QGIS3 python console
# >>> processing.algorithmHelp("qgis:distancematrix") 
# to get the object types it accepts for each parameter

inputlayer = QgsVectorLayer('/path/to/myinputfile.shp', 'layer 1', 'ogr')
targetlayer = QgsVectorLayer('/path/to/mytargetfile.shp', 'layer 2', 'ogr')

#do an Input Layers Validity Check
print(inputlayer.isvalid())
print(targetlayer.isvalid())

inputfield = QgsProperty.fromValue("InputPointID")
targetfield = QgsProperty.fromValue("TargetPointID")

matrixtype = QgsProperty.fromValue(0)
nearestpoints = QgsProperty.fromValue(0)

outputlayer = QgsProcessingOutputLayerDefinition('/path/to/myoutputfile.csv')

params = { 
    'INPUT' : inputlayer,
    'INPUT_FIELD' : inputfield,
    'TARGET' : targetlayer,
    'TARGET_FIELD' : targetfield,
    'MATRIX_TYPE' : matrixtype,
    'NEAREST_POINTS' : nearestpoints,
    'OUTPUT' : outputlayer
}

feedback = QgsProcessingFeedback()

res = processing.run('qgis:distancematrix', params, feedback=feedback)
res['OUTPUT'] # Access your output layer
Improve this answer
2
  • Don't forget to click the green-faded tick on the left-hand side of your answer to accept it. Also, it would be great if you could provide a second answer (or edit this one) to include a script which runs in parallel :)
    – Joseph
    Jul 30, 2019 at 10:24
  • 1
    Hi Joseph, see second answer as per your request
    – alanajones
    Aug 1, 2019 at 19:12
3
+200

As per Joseph's request please find attached alternative version to allow the script to run in GNU Parallel. It assumes QGIS3 and Debian.

The method I have found best is to design it as follows:

  1. convert the internal file calls in the python code to arguments

  2. create an OS script wrapper (e.g. bash) to run the python code (to make sure it can be run without problems via the shell before moving to GNU Parallel)

  3. create a GNU Parallel wrapper which runs the OS shell wrapper

I have the Distance Matrix working in parallel now. So following the above method, I adjusted the python code to the below:


######### INITIALISE QGIS STANDALONE ################

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer,
     QgsField,
     QgsFields,
     QgsProperty,
     QgsProcessingFeatureSourceDefinition,
     QgsProcessingOutputLayerDefinition
)

#start QGIS instance without GUI
#Make sure the prefix is correct. Even though QGIS is in '/usr/share/qgis',
#the prefix needs to be '/usr' (assumes Debian OS)

QgsApplication.setPrefixPath('/usr', True)
myqgs = QgsApplication([], False)
myqgs.initQgis()

######### INITIALISE THE PROCESSING FRAMEWORK ################

# Append the path where processing plugin can be found (assumes Debian OS)
sys.path.append('/usr/share/qgis/python/plugins')

#import modules needed
import processing
from processing.core.Processing import Processing

#start the processing module
processing.core.Processing.Processing.initialize()

######### Set Distance Matrix plugin parameters ############################

# I used this command in the QGIS3 python console
# >>> processing.algorithmHelp("qgis:distancematrix") 
# to get the object types it accepts for each parameter

inputlayer = qgis.core.QgsProperty.fromValue(sys.argv[1])
targetlayer = qgis.core.QgsProperty.fromValue(sys.argv[2])

inputfield = QgsProperty.fromValue(sys.argv[3])
targetfield = QgsProperty.fromValue(sys.argv[4])

matrixtype = QgsProperty.fromValue(0)
nearestpoints = QgsProperty.fromValue(0)

outputlayer = QgsProcessingOutputLayerDefinition(sys.argv[5])

params = { 
    'INPUT' : inputlayer,
    'INPUT_FIELD' : inputfield,
    'TARGET' : targetlayer,
    'TARGET_FIELD' : targetfield,
    'MATRIX_TYPE' : matrixtype,
    'NEAREST_POINTS' : nearestpoints,
    'OUTPUT' : outputlayer
}

feedback = QgsProcessingFeedback()

res = processing.run('qgis:distancematrix', params, feedback=feedback)
res['OUTPUT'] # Access your output layer

Then I created an OS wrapper for the python script. An example bash script is below:

#!/bin/bash

########### set the arguments for the Distance Matrix script (Input File, Target File, Input File ID Field, Target File ID Field) ################################
########### this script assumes there is 1 or more input files and the same target file  for all input file(s) #########################################

firstarg=$1 
firstargfileformat=".shp"

secondarg="path/to/targetfile"
secondargfileformat=".shp"

thirdarg="InputPointID"
fourtharg="TargetPointID"

fifthargfileprefix="YourChosenPrefix"
fifthargfileformat=".csv"

######## make a copy of the target file to use for processing which matches the name of the input file, for when it runs in parallel mode, so multiple parallel threads are not fighting for access to the same file ###############

#first get the name of the input file (without its path and without its extension)
inputfilenamewithoutfilepath=$(basename -- "$firstarg")
inputfilenamewithoutextension=$(basename --suffix=$firstargfileformat "$inputfilenamewithoutfilepath")

#test it has got the file name correctly
echo $inputfilenamewithoutextension

#then get the file extension of the target file
targetfilenamewithoutfilepath=$(basename -- "$secondarg")
targetfilenamewithoutextension=$(basename --suffix=$secondargfileformat "$targetfilenamewithoutfilepath")

#create the name of the temporary target file from the name of the input file (without its extension)
combinedtargetfilename=${inputfilenamewithoutextension}$secondargfileformat
combinedtargetfilenamewithpath=TempTargets/$combinedtargetfilename

#check the target file name (with and without path) has been derived correctly
echo "Combined Target File Name: $combinedtargetfilename Combined Target File Name (with path): $combinedtargetfilenamewithpath"

#copy the file
cp $secondarg $combinedtargetfilenamewithpath

#derive the outputfilename
combinedoutputfilenamepart1=${inputfilenamewithoutextension}$fifthargfileformat
combinedoutputfilename=${fifthargfileprefix}$combinedoutputfilenamepart1
combinedoutputfilenamewithpath=path/to/$combinedoutputfilename

#check the output file name (with and without path) has been derived correctly
echo "Combined Output File Name $combinedoutputfilename Combined Output File Name with path $combinedoutputfilenamewithpath"

#########run the standalone QGIS Distance Matrix ###############

python3 pyqgisstandalonedistancematrix.py $firstarg $combinedtargetfilenamewithpath $thirdarg $fourtharg $combinedoutputfilenamewithpath

######## delete temporary files ###############

rm $combinedtargetfilenamewithpath

####### exit the script############

exit 0

Then created a GNU Parallel wrapper. Example in bash script again.


#!/bin/bash

#reset the shell SECONDS timer
SECONDS=0

#log the start datetime
startdatetime=$(date)
echo "$startdatetime"

#run the parallel script
parallel ./OS_Wrapper_for_Distance_Matrix.sh ::: Inputs/*.shp

#get the number of seconds which passed since the script started
runduration=$SECONDS

#log the finish datetime
enddatetime=$(date)
echo "$enddatetime"

#elapsedsecs=$(( $(date -d "$enddatetime" "+%s")-$(date -d "$startdatetime" "+%s") ))

printf 'This run took %dd:%dh:%dm:%ds\n' $(($SECONDS/86400)) $(($SECONDS%86400/3600)) $(($SECONDS%3600/60)) \ $(($SECONDS%60))
echo "Run started at $startdatetime"
echo "Run finished at $enddatetime"

exit 0

That's it. you can add more arguments to the python and bash scripts if you need to for your particular needs. For the job I used it for over the last couple of days it went from taking just over 4 days using pyqgis via the GUI, to 5 hours using the PYQGIS standalone script with GNU Parallel on my 8-core machine.

Improve this answer
1
  • You can put the OS wrapper in a bash function and thus have it in the same file as the GNU Parallel wrapper script. Also instead of computing combined filenames in bash it may be easier to use GNU Parallel's replacement strings, and grab those as arguments in the function ($1, $2, $3...).
    – Ole Tange
    Aug 4, 2019 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.