I want to iterate through all integer type fields in a layer and generate new "display" fields marked with a 'd' at the end of the name using the appropriate field calculator expression.

The script basically works, it generates the correct columns and even populates them with correctly formatted strings, but it runs extremely slow on my reasonably fast machine and after it has run some time, it leaves most of the cells in the new columns empty (NULL), so it might have run into some sort of timeout.

If I perform the task manually via the Field calculator, it is done in a second. Perhaps a more experienced QGIS user could give me a hint about what I'm doing wrong here, it would by great: (QGIS Version used: 3.20.1 on Windows 10)

from qgis.core import QgsProject, QgsField, QgsExpression, QgsFeature
import processing
from PyQt5.QtCore import QVariant

layer = iface.activeLayer()

# Get name and type of attributes in layer
layerfields = [
    {"name": i.name(), "type": i.typeName()}
    for i in layer.fields()

# Filter for integer type fields 
intfields = list(filter(lambda x: x['type'].startswith('Integer'), layerfields))
intfieldsnames = list([x['name'] for x in intfields])

# Generate formatted string type field for display purposes via QgsExpression

prov = layer.dataProvider()
for a in intfieldsnames:
    fld = QgsField(a+'d', QVariant.String)
    idx = layer.fields().lookupField(a+'d')
    e = QgsExpression('format_number('+a+',0,\'de\')')
    c = QgsExpressionContext()
    s = QgsExpressionContextScope()
    x = layer.getFeatures()

    for f in x:
        value = e.evaluate(c)
        atts = {idx: value}
    x = layer.getFeatures()


Is this code faster (tested on a 300 features memory layer) :

layer = iface.activeLayer()

# Get name and type of attributes in layer
intfieldsnames = [i.name() for i in layer.fields() if i.type() in [QVariant.LongLong, QVariant.Int]]

with edit(layer):
    prov = layer.dataProvider()
    new_fields = [QgsField(f"{a}d", QVariant.String) for a in intfieldsnames]

    for feat in layer.getFeatures():
        for field_name in intfieldsnames:
            feat[f"{field_name}d"] = f"{feat[field_name]:,}".replace(",", ".")

  • Thank you! This works fine and gives me a string version of the integer field, but not a formatted one like with the field calculator expression format_number, which would give me points as thousand separators like: 1163235 -> 1.163.235. I don't know where to fit the field calculator expression.
    – gisnogud
    Sep 4 at 12:52
  • @gisnogud: see my edit, it works for me. Sep 4 at 13:12
  • Did the trick! Thank you! (So the takeaway is: Don't try to use field calculator expressions in pyqgis code...)
    – gisnogud
    Sep 4 at 13:23

You can also use Refactor fields. Takes ~15 s for a feature class with 44.000 features with 15 integer fields:

lyr = QgsProject.instance().mapLayersByName('sksNaturvardsavtal')[0]

#List all fields current mapping (a list of dictionaries, one dict for each field)
default_mapping = [{'expression':f.name(), 'length':f.length(), 'name':f.name(), 'precision':f.precision(), 'type': f.type()} for f in lyr.fields()]

#Find the integer fields
intfields = [f for f in lyr.fields() if 'int' in f.typeName().lower()]

#Create field mapping to add them as string fields, note the expression
newfields = [{'expression': 'format_number("{0}", 3)'.format(f.name()), 'length': f.length(),'name': '{0}_d'.format(f.name()), 'precision': 0,'type': QVariant.String} for f in intfields]

newmap = default_mapping+newfields

#Use refactor field to add and calculate the new string fields
processing.runAndLoadResults("native:refactorfields", {'INPUT':lyr,'FIELDS_MAPPING':newmap,'OUTPUT':'TEMPORARY_OUTPUT'})

(I ran the tool manually, Ctrl+Alt+H, copied the command syntax and adapted it using List Comprehension)

  • 2
    Seems to be really fast ! Sep 5 at 8:20
  • 1
    Wow. Thank you for your help! It's greatly appreciated.
    – gisnogud
    Sep 6 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.