19

What triggers the release of file locks in PyQGIS?

I am trying to delete a few data sources (used temporarily) by calling QgsVectorFileWriter.deleteShapeFile, but I have to quit QGIS before I can do that. I have loaded the sources into QgsVectorLayer objects.

Must all of these objects and references to them be garbage collected before I can delete the source?

Is there a way to force this?

I've managed to create a minimal code sample that fails. Make sure temp dir is empty before running.

from qgis.core import *
import processing, os, gc

project_temp_dir = "C:/Path/To/My/Dir/"      
layer1_path = project_temp_dir + "layer1.shp"
layer2_path = project_temp_dir + "layer2.shp"
input_layer = QgsMapLayerRegistry.instance().mapLayersByName('in_layer')[0]
if not input_layer.isValid(): raise Exception("Failed to grab input layer")

# Create layer 1
err = QgsVectorFileWriter.writeAsVectorFormat(input_layer, layer1_path, "utf-8", input_layer.crs())   
if err != QgsVectorFileWriter.NoError: raise Exception("Failed to write layer 1")

# Load layer 1
layer1 = QgsVectorLayer(layer1_path, "lyr1", "ogr")
if not layer1.isValid(): raise Exception("Failed to load layer 1")

# Use layer 1 to create layer 2, read-only makes no difference
# if not layer1.setReadOnly(): raise Exception("Could not set layer 1 to read-only")
processing.runalg("qgis:reprojectlayer", layer1, "EPSG:54030", layer2_path)

# Load layer 2
layer2 = QgsVectorLayer(layer2_path, "lyr2", "ogr")
if not layer2.isValid(): raise Exception("Failed to load layer 2")

del layer1
del layer2 
del input_layer
gc.collect()
print "Garbage: " + str(gc.garbage) # Empty

# Remove data sources for layers - FAILS!!
for f in os.listdir(project_temp_dir):          
    if f.endswith(".shp") and not os.path.isdir(project_temp_dir + f):              
        if not QgsVectorFileWriter.deleteShapeFile(project_temp_dir + f):
            # F*%&ing locks. 
            print "Failed to clear project temp directory."

I found that it works if I use QgsVectorFileWriter to create layer2, instead of the processing algorithm. I get the same error if try the qgis:clip algorithm.

Is this a bug in processing?

Am I using it wrong?

Improve this question

3 Answers 3

Reset to default
11

As it turns out, it works well if you add the layer to the map registry, and then remove it again. The map registry takes ownership of the layer, so when it is deleted from the registry, the locks are freed. Note that you have to add the layer to the legend (.addMapLayer(layer, addToLegend = False) won't work).

Still not sure whether to call this a solution or a workaround, but it does the job.

# ...

# Replace the following code (note: should do error checking on map registry functions):

# Load layer 1
layer1 = QgsVectorLayer(layer1_path, "lyr1", "ogr")
if not layer1.isValid(): raise Exception("Failed to load layer 1")
QgsMapLayerRegistry.instance().addMapLayer(layer1) #!!!!

# Use layer 1 to create layer 2  
processing.runalg("qgis:reprojectlayer", layer1, "EPSG:54030", layer2_path)

# Load layer 2
layer2 = QgsVectorLayer(layer2_path, "lyr2", "ogr")
if not layer2.isValid(): raise Exception("Failed to load layer 2")
QgsMapLayerRegistry.instance().addMapLayer(layer2) #!!!!

# Remove layer references
QgsMapLayerRegistry.instance().removeMapLayer(layer1.id()) #!!!!
QgsMapLayerRegistry.instance().removeMapLayer(layer2.id()) #!!!!

# Remove data sources for layers
for f in os.listdir(project_temp_dir):          
    if f.endswith(".shp") and not os.path.isdir(project_temp_dir + f):    
    # ...

If anyone has more info, I'd be happy to learn more about this.

Improve this answer
0
4

There are multiple file locks that all need to be removed. If I wanted to delete a file after the process was done with it, I had to release three separate locks:

(specific to the issue occurring within a processing algorithm, QGIS 3)

  1. QgsProject.instance().removeMapLayer(layer.id()) to get them out of the project
  2. del layer to remove the variable
  3. context.temporaryLayerStore().removeAllMapLayers() to remove the algorithm's hold on this (and any other) layers - I called this in postProcessAlgorithm to be sure it wouldn't interrupt anything else.

With a concrete example:

class ExProcessingAlgorithm(QgsProceessingAlgorithm):

  ...

  def processAlgorithm(self, parameters, context, feedback):

    ...

    layer1 = processing.run(...,
      is_child_algorithm=True, context=context, feedback=feedback)['OUTPUT']
    layer2 = QgsVectorLayer(path)

    # do something with both layers - now their file isn't needed anymore

    QgsProject.instance().removeMapLayers([layer1, layer2.id()])
    del layer1, layer 2

    return {'OUTPUT': ...}

  def postProcessAlgorithm(self, context, feedback):
    context.temporaryLayerStore().removeAllMapLayers()
    os.remove(self.output_path)  # <- the file layer1 and layer2 were both saved in
    return super().postProcessAlgorithm(context, feedback)
Improve this answer
1

The workaround presented by Oystein didn't work for me, and I was also not able to find an actual solution to the file lock problem. However, i was able to find another general workaround for the problem using the tempfile lib, which is the following:

Instead of having QGIS save layers (both raster and vector) on a predefined folder i used tempfile lib to create a temporary folder for those intermediate layers everytime i call the function that creates those layers (a different temporary folder will be created everytime you call the function). I did it like this:

import tempfile

tempfolder = tempfile.mkdtemp()

You can easily check the path to the temporary folder with:

print tempfolder

Now everytime you create a new Vector or Raster layer either as the result of a processing algorithm, or via the QgsRasterFileWriter or QgsVectorFileWriter you can set the OUTPUT parameter to something similar to:

raster_Path = (tempfolder + "/yourRasterName.tif")
vector_Path = (tempfolder + "/yourVectorName.shp")

If you put those lines into some function you can repeatedly call it without having to worry with excluding intermediate layers.

Another option to avoid creating intermediate layers in the first place is to define the OUTPUT parameter of processing algorithms to None, so that the output stays in the memory, and then define your new layers refering to that output with the "OUTPUT" key. Like so:

cutProcess = processing.runalg('gdalogr:cliprasterbymasklayer',
              someRasterLayer,  # INPUT <ParameterRaster>
              someMaskLayer,  # MASK <ParameterVector>
              "-9999",  # NO_DATA <ParameterString>
              False,  # ALPHA_BAND <ParameterBoolean>
              False,  # CROP_TO_CUTLINE <ParameterBoolean>
              False,  # KEEP_RESOLUTION <ParameterBoolean>
              5,  # RTYPE <ParameterSelection>
              4,  # COMPRESS <ParameterSelection>
              1,  # JPEGCOMPRESSION <ParameterNumber>
              6,  # ZLEVEL <ParameterNumber>
              1,  # PREDICTOR <ParameterNumber>
              False,  # TILED <ParameterBoolean>
              0,  # BIGTIFF <ParameterSelection>
              False,  # TFW <ParameterBoolean>
              "",  # EXTRA <ParameterString>
              None) # OUTPUT <OutputRaster>

 outputRasterLayer = QgsRasterLayer(cutProcess["OUTPUT"],
                            'newLayer')

I'm using QGIS 2.18.22 and consequently Python 2.7.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.