Iterating through fields and generating formatted string output

Question

I want to iterate through all integer type fields in a layer and generate new "display" fields marked with a 'd' at the end of the name using the appropriate field calculator expression.

The script basically works, it generates the correct columns and even populates them with correctly formatted strings, but it runs extremely slow on my reasonably fast machine and after it has run some time, it leaves most of the cells in the new columns empty (NULL), so it might have run into some sort of timeout.

If I perform the task manually via the Field calculator, it is done in a second. Perhaps a more experienced QGIS user could give me a hint about what I'm doing wrong here, it would by great: (QGIS Version used: 3.20.1 on Windows 10)

from qgis.core import QgsProject, QgsField, QgsExpression, QgsFeature
import processing
from PyQt5.QtCore import QVariant


layer = iface.activeLayer()

# Get name and type of attributes in layer
layerfields = [
    {"name": i.name(), "type": i.typeName()}
    for i in layer.fields()
    ]

# Filter for integer type fields 
intfields = list(filter(lambda x: x['type'].startswith('Integer'), layerfields))
intfieldsnames = list([x['name'] for x in intfields])

# Generate formatted string type field for display purposes via QgsExpression

layer.startEditing()
prov = layer.dataProvider()
for a in intfieldsnames:
    fld = QgsField(a+'d', QVariant.String)
    prov.addAttributes([fld])
    layer.updateFields()
    idx = layer.fields().lookupField(a+'d')
        
    e = QgsExpression('format_number('+a+',0,\'de\')')
    c = QgsExpressionContext()
    s = QgsExpressionContextScope()
    s.setFields(layer.fields())
    c.appendScope(s)
    e.prepare(c)
    
    x = layer.getFeatures()

    for f in x:
        c.setFeature(f)
        value = e.evaluate(c)
        atts = {idx: value}
        layer.dataProvider().changeAttributeValues({f.id():atts})
    x = layer.getFeatures()
layer.commitChanges()

Answer 1

Is this code faster (tested on a 300 features memory layer) :

layer = iface.activeLayer()

# Get name and type of attributes in layer
intfieldsnames = [i.name() for i in layer.fields() if i.type() in [QVariant.LongLong, QVariant.Int]]

with edit(layer):
    prov = layer.dataProvider()
    new_fields = [QgsField(f"{a}d", QVariant.String) for a in intfieldsnames]
    prov.addAttributes(new_fields)
    layer.updateFields()

    for feat in layer.getFeatures():
        for field_name in intfieldsnames:
            feat[f"{field_name}d"] = f"{feat[field_name]:,}".replace(",", ".")

        layer.updateFeature(feat)

Answer 2

You can also use Refactor fields. Takes ~15 s for a feature class with 44.000 features with 15 integer fields:

lyr = QgsProject.instance().mapLayersByName('sksNaturvardsavtal')[0]

#List all fields current mapping (a list of dictionaries, one dict for each field)
default_mapping = [{'expression':f.name(), 'length':f.length(), 'name':f.name(), 'precision':f.precision(), 'type': f.type()} for f in lyr.fields()]

#Find the integer fields
intfields = [f for f in lyr.fields() if 'int' in f.typeName().lower()]

#Create field mapping to add them as string fields, note the expression
newfields = [{'expression': 'format_number("{0}", 3)'.format(f.name()), 'length': f.length(),'name': '{0}_d'.format(f.name()), 'precision': 0,'type': QVariant.String} for f in intfields]

newmap = default_mapping+newfields

#Use refactor field to add and calculate the new string fields
processing.runAndLoadResults("native:refactorfields", {'INPUT':lyr,'FIELDS_MAPPING':newmap,'OUTPUT':'TEMPORARY_OUTPUT'})

(I ran the tool manually, Ctrl+Alt+H, copied the command syntax and adapted it using List Comprehension)