Removing vector layers with geopackage source with PyQGIS and getting win 32 error

Question

I'm aware that there are quite a few answers out there that have to do with removing datasources that are used to populate QGIS layers, but I feel like my question is slightly different in that it doesn't have to do with processing algorithms creating the problem but rather (I suspect) something in my approach.

I'm writing a plugin for QGIS 3.16.1 and I am creating layers and groups programmatically like this:

def create_vector_layer(source_path, style=None, layer_name=None, type="ogr"):
    try:
        if not layer_name:
            layer_name = Path(source_path).stem
        vlayer = QgsVectorLayer(source_path,
                            layer_name,
                            type)
        if not vlayer.isValid():
            raise Exception(f"Layer created is invalid {source_path}")
        if style:
            connect_layer_style(vlayer, style)
        return vlayer
    except Exception as e:
        raise e from None

After that I am adding them to the project with a call to this function:

def add_layer_to_project(layer, group_lst=None, add=True):
    """
    group name has to be list relative to root of project
    root
        group
            subgroup
                subsubgroup for example
    """
    try:
        # Add layer to Qgs instance
        root = QgsProject.instance().layerTreeRoot()
        QgsProject.instance().addMapLayer(layer, False)
        # If group is specified, create it if it doesn't exist yet
        if add:
            if group_lst:
                # Last item in list represent group to add layer to
                group = ensure_group(group_lst, root)
                group.addLayer(layer)
            else:
                root.addLayer(layer)
    except Exception as e:
        raise e from None

Just to make sure all information is included, this is the ensure group function:

def ensure_group(groups, root=None):
    """
    Function takes list of group names, first one has to be top level
    compared to root
    group
        subgroup1
            sub_subgroup
    etc as a list
    """
    print(f"groups {groups}")
    if root == None:
        root = QgsProject.instance().layerTreeRoot()
    found = False
    if groups:
        for child in root.children():
            if isinstance(child, QgsLayerTreeGroup):
                # If group exists, go check it's children
                if child.name() == groups[0]:
                    return ensure_group(groups[1:], child)
                    found = True
        if found == False:
            group = root.addGroup(groups[0])
            return ensure_group(groups[1:], group)
    else:
        return root

To clarify: I don't always add layers to the canvas, sometimes I only use them later on te add virtual layers based on them to the canvas. Not relevant per se but I figured I'd explain.

Now I need to delete certain groups, the layers contained in these groups that were generated automatically and their datasources (almost always geopackages).

I tried this:

def remove_group_and_children(group, generated_layers):
    root = QgsProject.instance().layerTreeRoot()
    # group = root.findGroup(group_name)
    if group is not None:
        for child in group.children():
            if isinstance(child, QgsLayerTreeGroup):
                remove_group_and_children(child, generated_layers)
# Only remove layer if it was generated automatically
            else:
                if child.name() in generated_layers:
                    try:
                        print(f"Removing layer {child.name()}")
                        QgsProject.instance().removeMapLayer(child.layerId())
                    except Exception as e:
                        print(e)
# Only remove group if it is empty
        if len(group.children()) == 0:
            root.removeChildNode(group)

This visually removes the correct groups and layers, but I can't remove the associated data sources until I restart QGIS entirely, only after restarting are the .gpkg-shm and .gpkg-wal files gone.

Now I'm not sure what is going wrong here. I've tried manually deleting the layers from the QGIS interface, I've tried every method I could think off from the Python console, but it seems like something is still 'open' for lack of a better word. I am 99% sure it's not a reference in my code, since I have tried uninstalling the entire plugin and then removing the layers as well. I can't see any layers after deletion with QgsProject.instance().mapLayers() and QgsProject.instance().layerTreeRoot() doesn't appear to have any children. I haven't found any place where a layer has a different memory address (I was thinking I may have accidentally copied somewhere)

I've found that this works from python console within QGIS:

layer = QgsVectorLayer(source_path, "layer", "ogr")
QgsProject.instance().addMapLayer(layer, False)
root = QgsProject.instance().layerTreeRoot()
group = root.addGroup('group')
group.addLayer(layer)
# Then to remove:
group_ref = QgsProject.instance().layerTreeRoot().findGroup('group')
for child in group_ref.children():
    QgsProject.instance().removeMapLayer(child.layerId())
if len(group_ref.children()) == 0:
    QgsProject.instance().layerTreeRoot().removeChildNode(group_ref)
os.remove(source_path)

What is the difference between my plugin approach and the script above?

Answer 1

Just in case anyone else makes the same mistake: the problem in my case was that I was creating my layers in a separate thread, not considering the fact that creating a layer creates a connection as well, which then went out of scope as soon as that thread was finished and thus stayed open. I have now moved the creation of layers into my main thread and everything works fine!