PyQGIS - Processing.runAndLoadResults(“qgis:mergevectorlayers”) throwing vague error, occasionally works

Question

I am trying to convert all the text files in a directory (CSV files that contain xy points with additional fields of associated data) into a single merged shapefile. Then create a line shapefile from that merged shapefile and load both into the current QGIS map I have open. I would like to have both the points and path shapefile present in the map.

I would consider myself an extreme Python novice so please take that into account.

My approach is as follows:

1. Convert all CSV files in a directory individually into shapefiles using QgsVectorFileWriter.
2. Merge all newly created shapefiles into one shapefile using the 'merge vector files' algorithm.
3. Create line file from newly created merged shapefile using 'points to path' algorithm.

Below is my latest iteration of the code. I apologize for including so much of the code, but I am not sure where the error lies.

import os, sys
from qgis.core import *
from processing.tools.vector import *

#target directory
root = "E:\\03_Data\\02_Onboard Deliverables\\02_Daily\\20200901\\06_Trackplots\\MAG\\1hz\\"
#add file prefix to root for uri
uriroot = "file:///"+ root 

#spatial reference variable
spatref = "epsg:6348"

#create list of source txt files.
directory_list = []
for f in os.listdir(root):
    if ".txt" in f:
        directory_list.append(f)
    else:
        pass

#set variables for x and y coordinates in txt files
xfield = "Grad Towfish CoG Easting"
yfield = "Grad Towfish CoG Northing"

#create file list for merge process to be populated with newly created shapefile paths
file_list = []

for f in directory_list:
    #Setup URI for vector layer creation
    uri = uriroot + f + "?encoding=%s&delimiter=%s&xField=%s&yField=%s&crs=%s" % ("UTF-8",",",xfield,yfield,spatref)
    
    #Make Vector layer object of txt file
    pointlayer = QgsVectorLayer(uri,os.path.split(f)[-1],"delimitedtext")
    
    #Setup file name for shapefile output
    filename = os.path.split(f)[-1]
    outfilename = root + filename.split('.')[0] + ".shp"
    
    #Append new shapefile path to file list for merge process
    file_list.append(outfilename)
    
    #Output shapefile to source file directory
    QgsVectorFileWriter.writeAsVectorFormat(pointlayer,outfilename,"UTF-8",driverName="ESRI Shapefile", layerOptions=['SHPT=Point'])
    
    #indicate success or failure of file creation
    filetest = QgsVectorLayer(outfilename, filename, "ogr")
    if filetest.isValid():
        print (filename + " successfully created")
    else:
        print (filename + " file creation failed")
        
mergefile = root + root[44:52] + "_mag_1hz_trackplots.shp"

#merge all shapefiles in directory and load result
processing.runAndLoadResults("qgis:mergevectorlayers",{"LAYERS": file_list, "OUTPUT": mergefile})

#create path shapefile from previously merged point shapefile and load result. 
processing.runAndLoadResults("qgis:pointstopath",{'DATE_FORMAT' : '', 'GROUP_FIELD' : None, 'INPUT' : mergefile, 'ORDER_FIELD' : 'Time', 'OUTPUT' : mergefile[:-3] + "_paths.shp"})

When the script fails the following error is returned.

Traceback (most recent call last): File "C:\PROGRA~1\QGIS3~1.10\apps\Python37\lib\code.py", line 90, in runcode exec(code, self.locals) File "", line 1, in File "", line 55, in File "C:/PROGRA~1/QGIS3~1.10/apps/qgis-ltr/./python/plugins\processing\tools\general.py", line 150, in runAndLoadResults context=context) File "C:/PROGRA~1/QGIS3~1.10/apps/qgis-ltr/./python/plugins\processing\core\Processing.py", line 181, in runAlgorithm raise QgsProcessingException(msg) _core.QgsProcessingException: There were errors executing the algorithm.

Some additional info:

• This code has worked at one time or another though I cannot figure out the correlation between when it is working and not. Once I changed the variable for the input list for the 'MergeVectorLayer' process and it worked but I have since tried that again it hasn't helped.
• I initially tried to find a way to recursively append each CSV to an exisiting shapefile (that I would create with the first CSV), negating the need for a merge, but I could not find a way to do that.

Update: This appears to be an issues with how I am passing the list to the 'Merge Vector Layer' algo. I proved this by manually inserting a list of the target files into the arguments and it has worked. That being said I have extensively investigated the list creation process and the output seems normal. As mentioned it also has worked from time to time so I am not sure why any given time would be different.

Answer 1

The issue ended up lying within one of the many files in the directory being iterated through. Issues with the text in one of the CSV files made the resulting shapefile have erroneous geometry and thus the 'Merge Vector Files' algorithm fail upon reaching it. Removed the file and the script works great! (for now).