4

Our QGIS project is dynamically generated on startup based on user roles. There are nearly 120 QgsVectorLayers that need to be generated. If created sequentially, the login process takes up to 2 minutes which is really annoying.

So, I thought I'll use a QThreadPool and create the layers in there. So, the threads basically just call this kind of code:

class LayerLoaderWorker(QRunnable):
    
    def run(self):
        db_connect = DatabaseConnection()
        uri = QgsDataSourceUri()
        uri.setConnection(db_connect.host, str(db_connect.port), db_connect.dbname, db_connect.user,
                          db_connect.password)
        uri.setDataSource('public', self.viewname, 'geometry', '', 'id')
        
        locker = QMutexLocker(mutex)
        self.layer = QgsVectorLayer(uri.uri(False), self.layerSettings.displayname, "postgres")
        self.finished = True
        self.loader.workerFinished()
        locker.unlock()

Only the layers are created in the threads, after creation they are added to the QgsProject in the main (UI) thread. Sometimes it seems to work, but most of the time the layers are not loaded correctly and the geometry column is empty.

Is the creation of the QgsVectorLayer object thread safe? It seems to me that this isn't the case. I read somewhere that UI-related things should be handled in the UI-Thread, but I thought creating the object (which seems to connect to the database which takes some time) is a non-ui-task.

Is it possible to load layers in parallel somehow?

Improve this question
11
  • Look at QgsTask. It is the usual method for background operations. The task shouldn't touch the GUI, as you have guessed.
    – wingnut
    Apr 27, 2021 at 15:44
  • I switched the layer loading to QgsTask, but unfortunately it still doesn't work correctly. I added the layer to the QgsProject in the finished()-Method of the task, since this method should be called from the main thread, as described here: docs.qgis.org/3.16/en/docs/pyqgis_developer_cookbook/tasks.html All layers are added to the mapTree, but most of them do not contain any features. I guess the creation of the QgsVectorLayer triggers stuff on the main thread or the dataprovider (postgres) is not thread safe. I guess I will give up, there seems to be no solution :-(
    – TheGrudge
    Apr 28, 2021 at 12:18
  • Always works for me, but I'm not using a database connection. There could be a timeout. I also use QgsTask.fromFunction() instead of subclassing. I declare the task global, as it had problems when I didn't. Creating the vector layer in the thread is ok.
    – wingnut
    Apr 28, 2021 at 12:41
  • I found one interesting thing: When I try to update / reload the layer from QGIS itself, it tells me that there are already too many clients connected. Maybe I need to tell the QgsVectorLayer (or QGIS itself) to disconnect after creation?
    – TheGrudge
    Apr 28, 2021 at 12:56
  • I guess you should probably do that anyway, especially in your loop.
    – wingnut
    Apr 28, 2021 at 13:01

1 Answer 1

Reset to default
1

Ok I'll give up, I tried everything I can think of. I deleted the object, I send the deleteLater signal to the metaObject of the task and so on. Nothing seems to work.

I used QThreads, QgsTasks as custom classes and now I tried to use QgsTask.fromFunction, nothing is working. As soon as the layer is returned, the db connection is never closed.

This is the example code I'm using right now:

@dataclass(frozen=True)
class LookupLoaderData:
    businessTypeId: int
    layer: QgsVectorLayer
    public_table: str


def loadLookupTableLayer(task, businessTypeId):
    display_name = BusinessModel().getObjectDisplayName(businessTypeId=businessTypeId)

    QgsMessageLog.logMessage('loading lookup layer name "{}"'.format(display_name),
                             MESSAGE_CATEGORY, Qgis.Info)

    uri = QgsDataSourceUri()
    uri.setConnection(db_connect.host, str(db_connect.port), db_connect.dbname,
                      db_connect.user, db_connect.password)
    uri.setDataSource(schema, public_table, None, None, None)
    uri.setWkbType(QgsWkbTypes.NoGeometry)
    layer = QgsVectorLayer(uri.uri(), display_name, "postgres")
    return LookupLoaderData(businessTypeId, layer, public_table)


def loadLookupTableLayerCompleted(exception, result: Optional[LookupLoaderData] = None):
    if not result or not result.layer:
        return
    oldName = result.layer.name()
    result.layer.setName(f'DEBUG ({result.businessTypeId}) {oldName}')
    result.layer.setReadOnly(True)
    layerTree = iface.layerTreeCanvasBridge().rootGroup()
    layer_group = layerTree.findGroup("Lookups")
    if not layer_group:
        layer_group = layerTree.insertGroup(0, "Lookups")
    layer_group.setItemVisibilityChecked(False)
    layer_group.setExpanded(False)
    QgsProject.instance().addMapLayer(result.layer, False)
    inserted_layer = layer_group.insertLayer(0, result.layer)
    inserted_layer.setItemVisibilityChecked(False)
    inserted_layer.setExpanded(False)

The connections stay open and the database will eventually refuse further connections. If I return something like this from the worker function:

return LookupLoaderData(businessTypeId, None, public_table)

The connections are closed because the object is killed, at least this is what I'm thinking :-) According to the time it takes to load all these layers, the tasks are actually doing something, but unfortunately I'm unable to tell because now I have no layer object that can be added to the project and the layerTree.

It seems to be impossible to load vector layers in the background, at least with a postgres data provider.

Improve this answer
5
  • You did db_connect.close() somewhere? Last ditch guess!
    – wingnut
    Apr 28, 2021 at 17:51
  • db_connect is just a dataclass with connection infos, only str and int values in there :-) I don't actively open the connection, this is all done by the dataProvider in the QgsVectorLayer
    – TheGrudge
    Apr 29, 2021 at 7:29
  • What DatabaseConnection library you using? pyodbc?
    – wingnut
    Apr 29, 2021 at 7:40
  • I only use a database library for reading some settings in our database, this is done by psycopg2 and sometimes QtSql for models that display and edit data in table views. But the QgsVectorLayer connects by its own driver, I don't do anything here, I'm only specifying the connection type which is "postgres", the rest if done by QGIS.
    – TheGrudge
    Apr 29, 2021 at 7:58
  • Try deleting the QgsDataSourceUri or setting it to None instead of the databse parameters. Or listening to some other unqualified commentator :-)
    – wingnut
    Apr 29, 2021 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.