2

I am pretty new to Python and QGIS but I am trying to create an algorithm for a processing plugin that will import a CSV of Tweet IDs. It then hydrates them using the Twarc Python library to get a CSV with extracted metadata from Twitter, and then create points from their coordinates in the CSV. I've gotten pretty far but now reached a wall.

When I run my processing plugin script QGIS completely crashes and doesn't give any error messages. The last bug I had was an error to do with QFileDialog so I suspect it is to do with that, but removing it still causes QGIS to crash.

Unfortunately, it's also part of a major project assignment for university due in two days, and the teaching staff are not overly helpful.

Do you have any advice specifically with QGIS crashing due to the script, or on the code in general?

I am using mac OS with QGIS 3.20 Odense.

My script:

#importing the modules I think I need

from qgis.PyQt.QtCore import QCoreApplication
from qgis.core import *

from twarc.client2 import Twarc2, expansions
import json
from twarc_csv import CSVConverter
from qgis.core import QgsCoordinateReferenceSystem
from PyQt5.QtGui import *
from PyQt5.QtWidgets import QApplication, QFileDialog



class HydrateGeoTweetsAlgorithm(QgsProcessingAlgorithm):
    

#processing boilerplate
    OUTPUT = 'OUTPUT'
    INPUT = 'INPUT'

    def initAlgorithm(self, config):

#importing csv of tweet IDs
        self.addParameter(
            QgsProcessingParameterFile(
                self.INPUT,
                self.tr('Input file')
            )
        )

#output point layer
        self.addParameter(
            QgsProcessingParameterFeatureSink(
                self.OUTPUT,
                self.tr('Output layer')
            )
        )

    def processAlgorithm(self, parameters, context, feedback):

#asking user to specify  a file to input
        qfd = QFileDialog()
        title = 'Open File'

        path = "/Users/"

        tweet_ids = QFileDialog.getOpenFileName(qfd, title, path)

#token you need to request from twitter API
        t = Twarc2(bearer_token="")

#iterating over csv using twarc to hydrate each tweet ID
        lookup = t.tweet_lookup(tweet_ids=tweet_ids)
        for page in lookup:
            result = expansions.flatten(page)
            with open("results.jsonl", "w+") as f:
                f.write(json.dumps(page) + "\n")

#converting jsonl to csv to get coordinates
        with open("results.jsonl", "r") as infile:
            with open("output.csv", "w") as outfile:
                convert = CSVConverter(infile, outfile, json_encode_all=False, json_encode_lists=True, json_encode_text=False, inline_referenced_tweets=True, allow_duplicates=False, batch_size=1000)
                convert.process()

#plotting points based on coordinates
        for tweet in convert:
            if tweet["geo.coordinates.coordinates"]:
                processing.run("native:createpointslayerfromtable",
                {'INPUT':'output.csv',
                'XFIELD':'cooridnates[1]',
                'YFIELD':'cooridnates[0]',
                'ZFIELD':'',
                'MFIELD':'',
                'TARGET_CRS':QgsCoordinateReferenceSystem(''),
                'OUTPUT':'\\Users\\tweet_points.shp'})

        return QgsMapLayerRegistry.instance().addMapLayers([tweet_points.shp])
            


#other processing boilerplate
    def name(self):

        return 'Hydrate and plot tweets'

    def displayName(self):

        return self.tr(self.name())

    def group(self):

        return self.tr(self.groupId())

    def groupId(self):

        return ''

    def tr(self, string):
        return QCoreApplication.translate('Processing', string)

    def createInstance(self):
        return HydrateGeoTweetsAlgorithm()
3
  • 1
    Remove any part of the code that is linked with QtResources and then try again. Such as Icon and etc...
    – Ash
    Oct 15, 2021 at 12:55
  • 1
    Thanks, I did that and I can run it again. Sorry for being clueless, but what should tweet_ids/the input file be assigned to then in order to iterate over it? I tried "tweet_ids = self.INPUT" and "tweet_ids = QgsProcessingParameterFile.INPUT" which caused another crash.
    – lil jakes
    Oct 15, 2021 at 13:24
  • 1
    Thanks, man, but @Ben_W has already answered your question thoroughly.
    – Ash
    Oct 16, 2021 at 6:48

1 Answer 1

3

This is a bit of a difficult question to give a good answer to because, although you have basically explained your intent in the question body, it is almost a 'wall of code', because unfortunately, the tweet hydrating logic is not easily reproducible. I for one have never used the twarc library, and it seems you need an api token which makes this difficult to test.

However, I will attempt to address the issues on the qgis processing side, as you have a few.

  1. You do not need QFileDialog at all- QgsProcessingParameterFile takes care of getting the input csv of tweet ids. What's more you can pass an extension argument so that only '.csv' files can be selected.

  2. To retrieve your input csv file and store it in the tweet_ids variable, you do this:

     tweet_ids = self.parameterAsFile(parameters, self.INPUT, 
     context)
    
  3. I don't think that you really want to run the child algorithm 'Create Points Layer From Table' iteratively in a for loop. It is difficult for me to say for sure because I don't know exactly what your csv looks like, but I guess that this will create a separate output layer for every tweet row in your csv. Instead just run it once and pass the csv file as input and the columns containing the X & Y values. Incidentally, the column names in your script are misspelt- cooridnates[1] and cooridnates[0]. Is that a typo in your script or a typo in the csv?

  4. Don't hard code a file path into the 'OUTPUT' parameter of the child algorithm. Again this is taken care of by the QgsProcessingParameterFeatureSink parameter which you added to your script in the initAlgorithm() method. If you just pass parameters['OUTPUT'], this allows the user to either save a file in any geospatial format or just create a temporary memory layer.

Additionally, this line:

return QgsMapLayerRegistry.instance().addMapLayers([tweet_points.shp])

is wrong for a few reasons. Firstly, QgsMapLayerRegistry was replaced by QgsProject in the QGIS 3 API. You probably copied that from an old answer or tutorial written for QGIS 2. Secondly, you don't need to worry about loading your output layer to the project. It is my understanding that this is handled by QgsProcessingContext. If the 'Open output file after running algorithm' checkbox is ticked, the output layer will be loaded on completion. Lastly, all qgis processing algorithms must return a dictionary object (even just an empty one) so you can just do: return {}.

However, if you want to be able to use an output layer from the algorithm as input for further processing, for example in a model or such, you can return a dictionary with a self.OUTPUT key and an output layer id value.

e.g. return {self.OUTPUT: dest_id}

(see end of script below for example)

And finally, all processing algorithms are run in a background thread by default. So if you do something which is not thread safe in the processAlgorithm() method, it can definitely cause QGIS to crash! To avoid this you can re-implement a flags() method in your algorithm class and return the No Threading flag e.g.

def flags(self):
    return super().flags() | QgsProcessingAlgorithm.FlagNoThreading

Below is a working example of an algorithm which takes an input csv file and runs native:createpointslayerfromtable as a child algorithm to create an output point layer. You still need to make sure that your twarc stuff is working because I can't really help you with that. If you have trouble with that part, I would suggest to get it working in the QGIS Python console, then paste and adapt it into the processing script.

Hopefully this example will help you:

from qgis.PyQt.QtCore import QCoreApplication, QVariant
from qgis.core import (QgsFeatureSink, QgsCoordinateReferenceSystem, QgsProcessing, QgsProcessingAlgorithm, QgsProcessingParameterFile, QgsProcessingParameterFeatureSink)
from qgis import processing
                    
                    
class ExAlgo(QgsProcessingAlgorithm):
    INPUT = 'INPUT'
    OUTPUT = 'OUTPUT'
 
    def __init__(self):
        super().__init__()
 
    def name(self):
        return "exalgo"
     
    def tr(self, text):
        return QCoreApplication.translate("exalgo", text)
         
    def displayName(self):
        return self.tr("Example script")
 
    def group(self):
        return self.tr("Examples")
 
    def groupId(self):
        return "examples"
 
    def shortHelpString(self):
        return self.tr("Example script with an input csv file")
 
    def helpUrl(self):
        return "https://qgis.org"
         
    def createInstance(self):
        return type(self)()
   
    def initAlgorithm(self, config=None):
        self.addParameter(QgsProcessingParameterFile(
            self.INPUT,
            self.tr("Input table"),
            extension='csv'))
            
        self.addParameter(QgsProcessingParameterFeatureSink(
            self.OUTPUT,
            self.tr("Output layer"),
            QgsProcessing.TypeVectorPoint))
 
    def processAlgorithm(self, parameters, context, feedback):
        
        # retrieve your input file parameter
        source = self.parameterAsFile(parameters, self.INPUT, context)
        
        # This is where you hydrate your tweet ids etc, convert to
        # .json and back to a csv with coordinate columns...
        
        point_layer_result = processing.run("native:createpointslayerfromtable",
                {'INPUT': source, # instead of source, pass your converted csv file
                'XFIELD':'Longitude', # make sure these fields exist & names are correct
                'YFIELD':'Latitude',
                'ZFIELD':'',
                'MFIELD':'',
                'TARGET_CRS':QgsCoordinateReferenceSystem('EPSG:4326'), # pass the appropriate crs
                'OUTPUT':parameters['OUTPUT']},
                is_child_algorithm=True,
                context=context,
                feedback=feedback)
 
        dest_id = point_layer_result['OUTPUT']

        return {self.OUTPUT: dest_id} # all processing algs must return a dictionary object (can be empty)

See short screencast below for results of this example script:

enter image description here

2
  • Thank you for this, it's an excellent response that cleared up a lot for me and I appreciate it greatly. I'm currently battling with a "PermissionError: [Errno 13] Permission denied: 'result'", but other than that your help really improved my code and understanding. Cheers.
    – lil jakes
    Oct 16, 2021 at 9:42
  • 1
    @lil jakes, no problem, you're welcome. Perhaps there is something helpful here: stackoverflow.com/questions/40984791/… For generic programming questions you are better off asking at Stack Overflow anyway as non-gis questions here usually get closed. Also, gis.stackexchange.com/help/someone-answers
    – Ben W
    Oct 16, 2021 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.