Speed of editing attributes in QGIS from a Python plugin

Question

I'm trying to edit the value of an attribute for each feature in a layer using a QGIS Python plugin. I've found that doing this outside of editing mode is much slower than while editing (even including committing the edits). See code below (lines interchangeable at the same point in a loop). The difference in speed for my sample dataset is 2 seconds (edit mode) vs 72 seconds (not edit mode).

Modifying an attribute in edit mode:

layer.changeAttributeValue(feature.id(), 17, QtCore.QVariant(value))

Modifying an attribute outside of edit mode:

layer.dataProvider().changeAttributeValues({ feature.id() : { 17 : QtCore.QVariant(value) } })

Is this an expected behavior? I don't need the user to be able to undo the changes, so I don't think I need to use edit mode.

Edit 1: See full code below with both versions included (but commented out):

def run(self):
    try:
        # create spatial index of buffered layer
        index = QgsSpatialIndex()
        self.layer_buffered.select()
        for feature in self.layer_buffered:
            index.insertFeature(feature)

        # enable editing
        #was_editing = self.layer_target.isEditable()
        #if was_editing is False:
        #    self.layer_target.startEditing()

        # check intersections
        self.layer_target.select()
        self.feature_count = self.layer_target.featureCount()
        for feature in self.layer_target:
            distance_min = None
            fids = index.intersects(feature.geometry().boundingBox())
            for fid in fids:
                # feature's bounding box and buffer bounding box intersect
                feature_buffered = QgsFeature()
                self.layer_buffered.featureAtId(fid, feature_buffered)
                if feature.geometry().intersects(feature_buffered.geometry()):
                    # feature intersects buffer
                    attrs = feature_buffered.attributeMap()
                    distance = attrs[0].toPyObject()
                    if distance_min is None or distance < distance_min:
                        distance_min = distance
                if self.abort is True: break
            if self.abort is True: break

            # update feature's distance attribute
            self.layer_target.dataProvider().changeAttributeValues({feature.id(): {self.field_index: QtCore.QVariant(distance_min)}})
            #self.layer_target.changeAttributeValue(feature.id(), self.field_index, QtCore.QVariant(distance_min))

            self.calculate_progress()

        # disable editing
        #if was_editing is False:
        #    self.layer_target.commitChanges()

    except:
        import traceback
        self.error.emit(traceback.format_exc())
    self.progress.emit(100)
    self.finished.emit(self.abort)

Both methods produce the same result, but writing via the data provider takes much longer. The function classifies proximity of building features to nearby fields (purple) using pre-created buffers (brown-ish).

Answer 1

The problem was, that each call to QgsDataProvider.changeAttributeValues() initiates a new transaction with all the related overhead (depending on the data provider and system configuration)

When the features are changed on the layer first (as in QgsVectorLayer.changeAttributeValue() ) all the changes are cached in memory, what is much faster and are then committed in one single transaction in the end.

The same buffering can be achieved within the script, (i.e. outside of the vector layer edit buffer) and then committed in one transaction by calling QgsDataProvider.changeAttributeValues() once, outside of the loop.

There is also a handy shortcut for this in recent QGIS versions:

with edit(layer):
    for fid in fids:
        layer.changeAttributeValue(fid, idx, value)