QGIS crashing on custom processing plugin

Question

I am pretty new to Python and QGIS but I am trying to create an algorithm for a processing plugin that will import a CSV of Tweet IDs. It then hydrates them using the Twarc Python library to get a CSV with extracted metadata from Twitter, and then create points from their coordinates in the CSV. I've gotten pretty far but now reached a wall.

When I run my processing plugin script QGIS completely crashes and doesn't give any error messages. The last bug I had was an error to do with QFileDialog so I suspect it is to do with that, but removing it still causes QGIS to crash.

Unfortunately, it's also part of a major project assignment for university due in two days, and the teaching staff are not overly helpful.

Do you have any advice specifically with QGIS crashing due to the script, or on the code in general?

I am using mac OS with QGIS 3.20 Odense.

My script:

#importing the modules I think I need

from qgis.PyQt.QtCore import QCoreApplication
from qgis.core import *

from twarc.client2 import Twarc2, expansions
import json
from twarc_csv import CSVConverter
from qgis.core import QgsCoordinateReferenceSystem
from PyQt5.QtGui import *
from PyQt5.QtWidgets import QApplication, QFileDialog



class HydrateGeoTweetsAlgorithm(QgsProcessingAlgorithm):
    

#processing boilerplate
    OUTPUT = 'OUTPUT'
    INPUT = 'INPUT'

    def initAlgorithm(self, config):

#importing csv of tweet IDs
        self.addParameter(
            QgsProcessingParameterFile(
                self.INPUT,
                self.tr('Input file')
            )
        )

#output point layer
        self.addParameter(
            QgsProcessingParameterFeatureSink(
                self.OUTPUT,
                self.tr('Output layer')
            )
        )

    def processAlgorithm(self, parameters, context, feedback):

#asking user to specify  a file to input
        qfd = QFileDialog()
        title = 'Open File'

        path = "/Users/"

        tweet_ids = QFileDialog.getOpenFileName(qfd, title, path)

#token you need to request from twitter API
        t = Twarc2(bearer_token="")

#iterating over csv using twarc to hydrate each tweet ID
        lookup = t.tweet_lookup(tweet_ids=tweet_ids)
        for page in lookup:
            result = expansions.flatten(page)
            with open("results.jsonl", "w+") as f:
                f.write(json.dumps(page) + "\n")

#converting jsonl to csv to get coordinates
        with open("results.jsonl", "r") as infile:
            with open("output.csv", "w") as outfile:
                convert = CSVConverter(infile, outfile, json_encode_all=False, json_encode_lists=True, json_encode_text=False, inline_referenced_tweets=True, allow_duplicates=False, batch_size=1000)
                convert.process()

#plotting points based on coordinates
        for tweet in convert:
            if tweet["geo.coordinates.coordinates"]:
                processing.run("native:createpointslayerfromtable",
                {'INPUT':'output.csv',
                'XFIELD':'cooridnates[1]',
                'YFIELD':'cooridnates[0]',
                'ZFIELD':'',
                'MFIELD':'',
                'TARGET_CRS':QgsCoordinateReferenceSystem(''),
                'OUTPUT':'\\Users\\tweet_points.shp'})

        return QgsMapLayerRegistry.instance().addMapLayers([tweet_points.shp])
            


#other processing boilerplate
    def name(self):

        return 'Hydrate and plot tweets'

    def displayName(self):

        return self.tr(self.name())

    def group(self):

        return self.tr(self.groupId())

    def groupId(self):

        return ''

    def tr(self, string):
        return QCoreApplication.translate('Processing', string)

    def createInstance(self):
        return HydrateGeoTweetsAlgorithm()

Answer 1

This is a bit of a difficult question to give a good answer to because, although you have basically explained your intent in the question body, it is almost a 'wall of code', because unfortunately, the tweet hydrating logic is not easily reproducible. I for one have never used the twarc library, and it seems you need an api token which makes this difficult to test.

However, I will attempt to address the issues on the qgis processing side, as you have a few.

1. You do not need QFileDialog at all- QgsProcessingParameterFile takes care of getting the input csv of tweet ids. What's more you can pass an extension argument so that only '.csv' files can be selected.

2. To retrieve your input csv file and store it in the tweet_ids variable, you do this:

    tweet_ids = self.parameterAsFile(parameters, self.INPUT, 
    context)
   

3. I don't think that you really want to run the child algorithm 'Create Points Layer From Table' iteratively in a for loop. It is difficult for me to say for sure because I don't know exactly what your csv looks like, but I guess that this will create a separate output layer for every tweet row in your csv. Instead just run it once and pass the csv file as input and the columns containing the X & Y values. Incidentally, the column names in your script are misspelt- cooridnates[1] and cooridnates[0]. Is that a typo in your script or a typo in the csv?

4. Don't hard code a file path into the 'OUTPUT' parameter of the child algorithm. Again this is taken care of by the QgsProcessingParameterFeatureSink parameter which you added to your script in the initAlgorithm() method. If you just pass parameters['OUTPUT'], this allows the user to either save a file in any geospatial format or just create a temporary memory layer.

Additionally, this line:

return QgsMapLayerRegistry.instance().addMapLayers([tweet_points.shp])

is wrong for a few reasons. Firstly, QgsMapLayerRegistry was replaced by QgsProject in the QGIS 3 API. You probably copied that from an old answer or tutorial written for QGIS 2. Secondly, you don't need to worry about loading your output layer to the project. It is my understanding that this is handled by QgsProcessingContext. If the 'Open output file after running algorithm' checkbox is ticked, the output layer will be loaded on completion. Lastly, all qgis processing algorithms must return a dictionary object (even just an empty one) so you can just do: return {}.

However, if you want to be able to use an output layer from the algorithm as input for further processing, for example in a model or such, you can return a dictionary with a self.OUTPUT key and an output layer id value.

e.g. return {self.OUTPUT: dest_id}

(see end of script below for example)

And finally, all processing algorithms are run in a background thread by default. So if you do something which is not thread safe in the processAlgorithm() method, it can definitely cause QGIS to crash! To avoid this you can re-implement a flags() method in your algorithm class and return the No Threading flag e.g.

def flags(self):
    return super().flags() | QgsProcessingAlgorithm.FlagNoThreading

Below is a working example of an algorithm which takes an input csv file and runs native:createpointslayerfromtable as a child algorithm to create an output point layer. You still need to make sure that your twarc stuff is working because I can't really help you with that. If you have trouble with that part, I would suggest to get it working in the QGIS Python console, then paste and adapt it into the processing script.

Hopefully this example will help you:

from qgis.PyQt.QtCore import QCoreApplication, QVariant
from qgis.core import (QgsFeatureSink, QgsCoordinateReferenceSystem, QgsProcessing, QgsProcessingAlgorithm, QgsProcessingParameterFile, QgsProcessingParameterFeatureSink)
from qgis import processing
                    
                    
class ExAlgo(QgsProcessingAlgorithm):
    INPUT = 'INPUT'
    OUTPUT = 'OUTPUT'
 
    def __init__(self):
        super().__init__()
 
    def name(self):
        return "exalgo"
     
    def tr(self, text):
        return QCoreApplication.translate("exalgo", text)
         
    def displayName(self):
        return self.tr("Example script")
 
    def group(self):
        return self.tr("Examples")
 
    def groupId(self):
        return "examples"
 
    def shortHelpString(self):
        return self.tr("Example script with an input csv file")
 
    def helpUrl(self):
        return "https://qgis.org"
         
    def createInstance(self):
        return type(self)()
   
    def initAlgorithm(self, config=None):
        self.addParameter(QgsProcessingParameterFile(
            self.INPUT,
            self.tr("Input table"),
            extension='csv'))
            
        self.addParameter(QgsProcessingParameterFeatureSink(
            self.OUTPUT,
            self.tr("Output layer"),
            QgsProcessing.TypeVectorPoint))
 
    def processAlgorithm(self, parameters, context, feedback):
        
        # retrieve your input file parameter
        source = self.parameterAsFile(parameters, self.INPUT, context)
        
        # This is where you hydrate your tweet ids etc, convert to
        # .json and back to a csv with coordinate columns...
        
        point_layer_result = processing.run("native:createpointslayerfromtable",
                {'INPUT': source, # instead of source, pass your converted csv file
                'XFIELD':'Longitude', # make sure these fields exist & names are correct
                'YFIELD':'Latitude',
                'ZFIELD':'',
                'MFIELD':'',
                'TARGET_CRS':QgsCoordinateReferenceSystem('EPSG:4326'), # pass the appropriate crs
                'OUTPUT':parameters['OUTPUT']},
                is_child_algorithm=True,
                context=context,
                feedback=feedback)
 
        dest_id = point_layer_result['OUTPUT']

        return {self.OUTPUT: dest_id} # all processing algs must return a dictionary object (can be empty)

See short screencast below for results of this example script: