2

I have a question regarding adding attribute fields on an existing vector file in pyQGIS. I’m writing a processing script for QGIS, and so far, I use the QGIS field calculator to add new attribute fields and then fill every feature with a value. The script looks something like this:

layer = QgsVectorLayer(myVectorFile,"", "ogr")
Field_1=processing.runalg('qgis:fieldcalculator',layer,'Field_1',1,1.0,1.0,True,Value_1,None)
Field_2=processing.runalg('qgis:fieldcalculator',Field_1['OUTPUT_LAYER'],'Field_2',1,1.0,1.0,True,Value_2,None)
Field_3=processing.runalg('qgis:fieldcalculator',Field_2['OUTPUT_LAYER'],'Field_3',1,1.0,1.0,True,Value_3,None)

Etc.

This works as expected, but it takes a very long time when I run the script, since it runs on very large vector files with a lot of features. I’m looking for a better/more performant way to add fields and file them with a value in pyQGIS. I imagen that maybe it would be possible to only edit the .dbf file? I’m open to suggestions.

  • You could hit the file directly with OGR in python. A bit more long winded but should be faster.. but if you really want to go faster use OGR in C# or C++ if you're on Windows. – Michael Stimson Apr 26 '18 at 12:10
  • Well, I think OGR in python could be a way to go, but I still don't know how ;) – Miron Apr 26 '18 at 12:18
2
from osgeo import ogr

############################################################
shapefile = 'your_shapefile.shp'
driver = ogr.GetDriverByName("ESRI Shapefile")
dataSource = driver.Open(shapefile, 1) # 1 means read/write
############################################################

layer = dataSource.GetLayer()

# I assume you wanna add integer fields
layer.CreateField(ogr.FieldDefn("Field_1", ogr.OFTInteger))
layer.CreateField(ogr.FieldDefn("Field_2", ogr.OFTInteger))
layer.CreateField(ogr.FieldDefn("Field_3", ogr.OFTInteger))

# integer values
value_1 = 11
value_2 = 22
value_3 = 33

for feature in layer:
    feature.SetField("Field_1", value_1)
    feature.SetField("Field_2", value_2)
    feature.SetField("Field_3", value_3)
    layer.SetFeature(feature)

If you want to save the changes in a new file, replace highlighted three lines of code into the following lines. (The shortest way is to copy-paste the shapefile)

from shutil import copyfile

shapefile = 'your_shapefile.shp'
new_shapefile = 'your_new_shapefile.shp'    
copyfile(shapefile, new_shapefile)

driver = ogr.GetDriverByName("ESRI Shapefile")
dataSource = driver.Open(new_shapefile, 1)

For further information, look at Python GDAL/OGR Cookbook.

  • Thanks, that works and it seems to be quite fast! Just one thing: right now, the script ist structured so that the Input file is edited and the changes are saved in the input file. What can I do to save the changes in a new file? Thanks a lot anyway! – Miron Apr 26 '18 at 14:18
  • If you just want to add some fields and save it as new file, the shortest and easiest way is to copy it. But If you don't want to add some input layer fields, plaese review Python GDAL/OGR Cookbook. – Kadir Şahbaz Apr 26 '18 at 15:01
  • Thanks for the additional help! This works well. The only thing is: for some input shape files, I get the error: "'NoneType' object has no attribute 'GetLayer' See log for more details" This error points at the line "layer = dataSource.GetLayer()". If I try the same file withe the old method, it workd withaot an error... – Miron Apr 26 '18 at 15:25
  • 1
    I've no idea. This post may help. – Kadir Şahbaz Apr 26 '18 at 15:30
  • Thanks for the Link, it helped me figuring out what caused the error: apparently, the ogr driver “ESRI Shapefile” fails to load the file if the .dbf contains umlauts. Such a pity. So I’m stuck with the old method. – Miron Apr 27 '18 at 7:47
1

You could try this (example), but I dont know if this provides you more performance.

    list = []
    field1 = QgsField("test1", QVariant.String)
    field2 = QgsField("test2", QVariant.String)
    list.append(field1)
    list.append(field2)
    layer.startEditing ()
    pr = layer.dataProvider()          
    pr.addAttributes(list)
    layer.updateFields()    
    features=layer.getFeatures() 
    i=0
    idx1=0
    idx2=0
    for f in features:
     if i == 0:
      idx1=f.fieldNameIndex("test1")
      idx2=f.fieldNameIndex("test2")         
      i=1
     layer.changeAttributeValue(f.id(), idx1, "value1")
     layer.changeAttributeValue(f.id(), idx2, "value2")
    layer.commitChanges()   
1

Miron, one way to do what you want to do is to break the job down into the two tasks: 1. Add field and 2. Populate field.

To add a field create a layer, enter editing mode, add the field(s), and commit the changes. Like:

# create layer
tl = QgsVectorLayer(input_directory + input_file, "bufflayer", "ogr")
QgsMapLayerRegistry.instance().addMapLayer(tl)
pr = tl.dataProvider()

# Enter editing mode
tl.startEditing()

# add fields
pr.addAttributes( [ QgsField("chainage", QVariant.Int),
            QgsField("buffer_rad",  QVariant.Int),
            QgsField("No_kills", QVariant.Int) ] )

# Commit changes
tl.commitChanges()

To populate your fields set a variable to hold the index of the field you are filling, open a vector file with writing enabled, select the feature you want to update, update the feature attribute in the chosen field, commit the changes. This process updates one feature at a time - you can loop through whichever features you want to populate. Like:

 #  set field index for output to stats shapefile
         if v_class == 1:
             fldINDEX = ps_code + 5
         elif v_class == 2:
             fldINDEX = ps_code + 20
         elif v_class == 3:
             fldINDEX = ps_code + 35
         elif v_class == 4:
             fldINDEX = ps_code + 50

         #  open the output shapefile for writng the vegetation metrics
         inShapefile = output_directory + statsout_buff_layer
         inDriver = ogr.GetDriverByName("ESRI Shapefile")  #  specify Shapefile driver
         inDataSource = inDriver.Open(inShapefile, 1)  #  NOTE - "1" required to open for writing
         slayer = inDataSource.GetLayer()

         #  set query string to select record where vegetation metrics will be written
         queryString = "%s%s" % ("XL_ID = ", ExcelID)

         #  select record 
         slayer.SetAttributeFilter(queryString)

         #  set output in appropriate field of selected feature
         for feature in slayer:
            feature.SetField(fldINDEX, attr_patch)
            slayer.SetFeature(feature)

         slayer.CommitTransaction()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.