2

I have a simple script for processing thousands of rasters/shapefiles. i am not able to delete temporary files, which are created by processing. I can pass them to:

'memory:output_file'

but not for all of them, because sometimes I need to use gdal, which does not know memory. So I create gpkg/shapefile. But after 64 files (which are hold in memory, I get error from sqlite because it can not hold more than 64 open files...)

Sqlite ERROR: Too many connections: max 64

I am using standalone QgsApplication and calling processing.

The problem is, somehow the PyQGIS is locking files after processing. Similar problem here: QGIS: Release file lock on file used in a processing algorithm at the end of script

Another one here: Releasing PyQGIS file locks?

Another one: https://issues.qgis.org/issues/12502 (which says fixed/implemented)

And many others. Their solution does not work for me. If i try to put it into:

QgsProject.instance().addMapLayer(layer)
QgsProject.instance().removeMapLayer(layer.id())
QgsProject.instance().mapLayers() ## which returns empty dict, all good here

And check there are no files in registry, there is still lock on files: gpkg.wal and gpkg.shm

So i am not able to delete the file with:

os.remove(file)

My only solution is (which is not good, since I would have to call it all the time):

QgsApplication.exitQgs()

After that, I can delete the file. But if I try to start the application again, Python crashes. Any ideas how to release the damn lock?

My script is eg:

def selectByLocation(input, predicate, intersect_lyr, output):
    process = "qgis:extractbylocation"
    params = {
        'INPUT' : input,
        'PREDICATE' : predicate,
        'INTERSECT' : intersect_lyr,
        'OUTPUT' : output
    }
    out = processing.run(process, params)
    return out["OUTPUT"]

def removeLowDepths(raster, threshold, rn, workspace = temp):
    workspace_memory = 'memory:'
    import uuid
    ## cant overwrite file because of lock, so i am using uuid to create many temp files, which i can not delete
    expression = "data < " + str(threshold)
    lowDepthMaskR =  setNull(raster, expression, os.path.join(workspace, "lowDepthMaskR.tif"))
    lowDepthMask = polygonize(lowDepthMaskR, workspace + f'//lowDepthMask_{str(uuid.uuid4())[:8]}.gpkg', epsg)
    fixed = fixGeom(lowDepthMask, workspace_memory +"lowDepthMask_fixed")
    lowDepthMaskSel = selectByLocation(fixed, 0, rn, workspace_memory + "lowDepthMaskSel")
    finalMask = selectByAttribute(lowDepthMaskSel, 'DN', 0, 1, workspace_memory + 'finalMask')

    return finalMask

Edit for comment:

def removeLowDepths(raster, threshold, rn, workspace = temp):
    workspace_memory = 'memory:'
    import uuid, time
    id = str(uuid.uuid4())[:8]
    expression = "data < " + str(threshold)
    lowDepthMaskR =  setNull(raster, expression, os.path.join(workspace, "lowDepthMaskR.tif"))
    lowDepthMask = polygonize(lowDepthMaskR, workspace + f'//lowDepthMask_{id}.gpkg', epsg)

    print(QgsProject.instance().mapLayers()) ## prints {} so empty dict
    del lowDepthMask ## se picture, still locked
    time.sleep(60)
    os.remove(workspace + f'//lowDepthMask_{id}.gpkg')
    return finalMask

enter image description here

  • Did you try the del methods for deleting the files after use? E.g. del lowDepthMaskSel before the line return finalMask? – Joseph Sep 6 at 13:18
  • Yes, I tried that before and now again, see update – Jan Doležal Sep 6 at 13:27
1

Unfortunately it seems like this issue can not be solved easily. So for me, it was easier to just switch to another tools and not use QGIS processing (which is a damn shame...)

So I rewrote most processing via GeoPandas, Shapely, GDAL, numpy and it is working as expected. I can delete files just fine now.

def select_by_location(select_from, overlay, output, method = 'intersect', driver='GPKG'):
    if not isinstance(select_from, gpd.GeoDataFrame):
        select_from = gpd.read_file(select_from)
    if not isinstance(overlay, gpd.GeoDataFrame):
        overlay = gpd.read_file(overlay)

    if method.lower() == 'intersect':   
        overlay = gpd.GeoDataFrame({'geometry': overlay['geometry'].values}, crs = overlay.crs)
        selection = gpd.sjoin(select_from, overlay, op='intersects')
        selection = check_geometry(selection)
        if output:
            selection.to_file(output, driver=driver)
        else:
            return selection

    del select_from, overlay
    return output

def check_geometry(input):
    from shapely.wkt import loads
    if not isinstance(input, gpd.GeoDataFrame):
        df = gpd.read_file(input)
    df = input
    df['geometry'] = df['geometry'].astype(str).apply(loads)
    return df

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.