About speed, commitChanges too slow. How avoid this?

Question

I am working on a standalone script, executed out of QGIS console, and I have noticed that the script takes a lot of time when it executes the line: layerB.commitChanges ()

How my script works ?

I have two files that are identical, one is the copy of another.

The script loads both files (fileA and fileB), then it searches for similar points (very near, distance <= 15m. ) over fileA, and deletes it over the fileB. When finished, do commit over the fileB

What would be the fastest way to do this task? I hava read this-> Is it possible to change the geometry without startEditing the layer But I do not finish understanding the proposed solution.

The input data table is approximately 200,000 points and the output table is almost half.

layerA = QgsVectorLayer("fileA.sqlite", "layerA", "ogr")
layerB = QgsVectorLayer("fileB.sqlite", "layerB", "ogr")

featA = [feat for feat in layerA.getFeatures()]
featB = [feat for feat in layerB.getFeatures()]

del_feat = []

target_spatial_index = QgsSpatialIndex()
# populate the spatial index
for f in featB:
    target_spatial_index.insertFeature(f)


with edit(layerB):
    for feat in featA:

        # Skip if into deleted list
        if feat.id() not in del_feat:

            point1 = feat.geometry().asPoint()

            nearest_ids = target_spatial_index.nearestNeighbor(point1,2)

            high_id=0
            for id in nearest_ids:

                outFeat = layerB.getFeatures(QgsFeatureRequest().setFilterFid(id)).next()

                #skip himself
                if outFeat.id() != feat.id():

                    # distance between points
                    point2 = outFeat.geometry().asPoint()
                    dist = sqrt(point1.sqrDist(point2))

                    high_id=outFeat.id()


                    if dist <= 15:
                        # remove from spatial index
                        target_spatial_index.deleteFeature(outFeat)

                        layerB.deleteFeatures([high_id])
                        # add to list of deleted
                        del_feat.append(high_id)


layerB.commitChanges()
crs=QgsCoordinateReferenceSystem("epsg:32629")
print "Saving file..."
# Save memory layer to file
error = QgsVectorFileWriter.writeAsVectorFormat(layerB, "fileB.sqlite", "UTF-8", crs , "spatialite")

Answer 1

I think that you may try using this code:

featA = [feat for feat in layerA.getFeatures()]
featB = [feat for feat in layerB.getFeatures()]

del_feat = []

target_spatial_index = QgsSpatialIndex()
# populate the spatial index
for f in featB:
    target_spatial_index.insertFeature(f)

for ft in featA:
    inGeom = ft.geometry()
    buff = inGeom.buffer(15, -1) # '15' is the searching distance, in meters
    idsList = target_spatial_index.intersects(buff.boundingBox())
    for id in idsList:
        if buff.intersects(inGeom) and id != ft.id():
            if id not in del_feat:
                del_feat.append(id)
                target_spatial_index.deleteFeature(ft)

# Delete ids from layerB
layerB.startEditing()
layerB.deleteFeatures(del_feat)
layerB.commitChanges()

Just some comments:

• I have used a buffer of 15 m instead of searching for the nearest neighbor (why you only searched for the nearest point? What happens in your code if there are more than one point around your current feature?);
• I followed your idea of deleting the ids from del_feat directly from layerB, but however I think it should be faster if you create a new memory layer instead of deleting features from an existing layer: in this way, you could avoid to use commitChanges().

I tested the code on a sample set of 20000 points:

• using my code, the duration time was 0:00:06.038000 (more or less 6 seconds);
• using your code, the duration time was 0:00:15.723000 (more or less 16 seconds).

I underline that the creation of a new layer should further reduce the execution time.

Finally, I suggest you to have a look on this very useful post for speeding up your scripts.

Answer 2

Thanks, HowToInQGIS, finally using an empty layerB in Memory and populating, not deleting, the script is enought fast, only 3m38s for 55.000 points. (I use a set, not list, to save element not added). I also avoid update spatial_index by eliminate the instruction 'target_spatial_index.deleteFeature(outFeat)'

featA = [feat for feat in layerA.getFeatures()]

# Memory Layer
layerB = QgsVectorLayer("Point?crs=epsg:32629&index=yes", "layerB", "memory")

# Add fields to memory layer
attr = layerA.dataProvider().fields().toList()
outData = layerB.dataProvider()
outData.addAttributes(attr)
layerB.updateFields()

cont_features=len(featA)

#Boolean array to control elements to skip
del_feat=[True]*(cont_features+1) 

target_spatial_index = QgsSpatialIndex()
# populate the spatial index
for f in featA:
    target_spatial_index.insertFeature(f)

for f in featA:

    # Skip if marked has false
    if del_feat[f.id()]:

        point1 = f.geometry().asPoint()

        nearest_ids = target_spatial_index.nearestNeighbor(point1,2)

        request = QgsFeatureRequest()
        request.setFilterFids(nearest_ids)

        features = layerA.getFeatures(request)

        for featNear in features:

            #Add himself to output Layer
            if featNear.id() == f.id():
               outData.addFeatures([featNear])
               countAdd += 1
            else:

                # distancia between points
                point2 = featNear.geometry().asPoint()
                dist = sqrt(point1.sqrDist(point2))

                # Add to del_feat if distance < 15m
                if dist <= 15:
                    # Mark to skip it
                    del_feat[featNear.id()]=False


layerB.updateExtents()
layerB.commitChanges()
print "Saving file..."
# Save memory layer to file
error = QgsVectorFileWriter.writeAsVectorFormat(layerB, "POI2.sqlite", "UTF-8", layerA.crs() , "SpatiaLite")
print "File saved."
if error == QgsVectorFileWriter.NoError:
    print "success! writing output file"