Run QGIS Processing Algorithms on Multiple Input Files Using a Standalone Script

Question

I'm new to QGIS(3.20.2), and have even less experience with python. I've been working on automating a simple task and have been having difficulty tying together the pieces of information that I've found on the forum. What I'm looking to do is create a standalone script that runs without opening QGIS, takes all the text files from a specified folder, and creates output polygon files from each of these text files and places them in another specified folder. So far, I've mostly been able to accomplish this task within the python console of QGIS, but am having trouble looping the processing algorithms so they run on all the files in a folder, and running this script outside of QGIS. My script so far does the following:

• Creates delimited text layers from all the text files in a folder and adds them to the map. The text file format is shown below:

ID	X	Y	Sort
1	-77.1	36.4	1
2	-77.1	36.3	1
3	-77.09	36.2	2
4	-77.09	36.3	2

• Runs the Point to Path algorithm on an individual input file using the start and end xy coordinates, which has field_4 (sort) as the path group expression.
• Runs the Polygonize algorithm on the input file from the previous step and exports it to a .kml file in a specified folder.

The code that I'm using to do this is as follows:

import os.path, glob
from qgis.core import QgsProject, QgsVectorLayer
layers=[]
for file in glob.glob('C:/Users/1134377/Documents/ASD/ModelBuilder/*.txt'): # Change this base path and change backslashes to forward slashes
  uri = "file:///" + file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:4326" % (",","field_2","field_3") #(Delimeter, XField, YField) #1st column in file is column 1
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.setFieldAlias(1,'X') #these order of columns go: 0, 1, 2 etc
  vlayer.setFieldAlias(2,'Y')
  layers.append(vlayer)

QgsProject.instance().addMapLayers(layers)
processing.runAndLoadResults("native:pointstopath", {'INPUT':'delimitedtext://file:///C:/Users/1134377/Documents/ASD/ModelBuilder\\*.txt?delimiter=,&xField=field_2&yField=field_3&useHeader=no&crs=epsg:4326','CLOSE_PATH':False,'ORDER_EXPRESSION':'','NATURAL_SORT':False,'GROUP_EXPRESSION':'\"field_4\"','OUTPUT':'C:/MyFolder/Path.shp'})
processing.runAndLoadResults("native:polygonize", {'INPUT':'C:/MyFolder/Path.shp','KEEP_FIELDS':False,'OUTPUT':'C:/Users/1134377/Documents/ASD/ModelBuilder/Outputs/Poly.kml'})

The limitation of this script is that it doesn't run the algorithms (point to path & polygonize) on all the text layers that are added to the map using the beginning portion of code. It also needs to be run within the python console within QGIS, and not as a standalone script.

Answer 1

Building on recipes from the two answers here and here, you can try the script below. You will need to edit at least a couple of lines. Depending on how you installed QGIS (whether you used the standalone installer or OsGeo4W installer) your QgsApplication prefix path and the path to the processing plugin may be different from what I used in the script.

Also, you will need to edit the in_folder and out_folder paths to match your own file system.

import sys

import os

from qgis.core import (QgsApplication, QgsVectorLayer)
from qgis.analysis import QgsNativeAlgorithms

# See https://gis.stackexchange.com/a/155852/4972 for details about the prefix 
QgsApplication.setPrefixPath('C:/OSGeo4W/apps/qgis', True)
qgs = QgsApplication([], False)
qgs.initQgis()

# Append the path where processing plugin can be found
sys.path.append('C:\\OSGeo4W\\apps\\qgis\\python\\plugins')

import processing
from processing.core.Processing import Processing
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

in_folder = 'C:\\Users\\Ben\\Desktop\\TEMP\\txt_files'

out_folder = 'C:\\Users\\Ben\\Desktop\\TEMP\\kml_files'

def save_as_kml(in_file, save_location):
    out_layer = os.path.join(save_location, f'{in_file.name.replace(".txt", ".kml")}')
    uri = 'file:///{}?delimiter={}&crs=epsg:4326&xField={}&yField={}'.format(in_file.path, '\\t','X', 'Y')

    vlayer = QgsVectorLayer(uri, os.path.basename(in_file), 'delimitedtext')
    paths = processing.run("qgis:pointstopath",
                {'INPUT':vlayer,
                'CLOSE_PATH':True,
                'ORDER_FIELD':'Sort',
                'GROUP_FIELD':'',
                'DATE_FORMAT':'',
                'OUTPUT':'TEMPORARY_OUTPUT'})
    processing.run("native:polygonize",
                {'INPUT':paths['OUTPUT'],
                'KEEP_FIELDS':False,
                'OUTPUT':out_layer})

src_dir = os.scandir(in_folder)
for file in src_dir:
    if file.name.endswith('.txt'):
        save_as_kml(file, out_folder)

I'm not sure how you run your standalone scripts, but if you have trouble, here is how I do it:

Save the script above as a .py file (e.g. save_as_kml.py)

Next, create a batch file with the following content:

@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
@echo off
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;C:\OSGeo4W\apps\Qt5\bin
path %PATH%;C:\OSGeo4W\apps\Python39\Scripts
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis
set GDAL_FILENAME_IS_UTF8=YES
rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python
set PYTHONHOME=%OSGEO4W_ROOT%\apps\Python39
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins

cmd.exe

Again, the OSGEO4W_ROOT may be different than mine depending on your installation, just make sure it points to your main installation directory which contains the bin folder.

Save the batch file in the same location as your .py file. Then you can just double click the batch to launch it and, at the prompt, type: python save_as_kml.py to run your script.