PyQGIS: Speed up for adding features and attribute arrays?

Question

Context: I'm writing a dynamic import tool for QGIS (2.2 to 2.5) with python. Datatype: Binary Datasets with environmental simulations from HPC which might contain millions of features and terabytes of attributes.

Current status: I wrote some python that extracts the features and attributes at a certain time step.

• Loading Data for 100000 features takes < 0.5s via python.
• Displaying them takes 30s.
• Data structure: Double precision arrays

Now my question in short: Is it possible to speed up the change of attributes? Can I change 10 attributes for 100 000 features in one step? Rather then doing 1000000 loops, one per attribute?

I know QGIS would be fast enough, because: If I save my imported data as shapefile and reload it, the shapefile reader takes only ~2.5sec for 1 000 0000 attributes.

Here is my current code:

#---------------------------------------------------------------
# Create features
#
#
vl = QgsVectorLayer("polygon", myFile + " | Triangles", "memory")      # New memory layer
pr = vl.dataProvider()                                                 # Dataprovider

# Enter editing mode
vl.startEditing()

#------
# Attribut col
pr.addAttributes( [ QgsField("ID",  QVariant.Int) ] )   

#------
# Paint Triangles
i = 0 
for TRI in mesh.ELE:
    i += 1
    feat = QgsFeature()
    ep1 = TRI.P1-1
    ep2 = TRI.P2-1
    ep3 = TRI.P3-1
   feat.setGeometry(QgsGeometry.fromPolygon([[ \ QgsPoint(mesh.HEAD.X_origin+mesh.x[ep1],mesh.HEAD.Y_origin+mesh.y[ep1]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep2],mesh.HEAD.Y_origin+mesh.y[ep2]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep3],mesh.HEAD.Y_origin+mesh.y[ep3]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep1],mesh.HEAD.Y_origin+mesh.y[ep1])]]))
    attrdict = [i,]
    feat.setAttributes(attrdict)
    (res, outFeats) = pr.addFeatures( [ feat ] )

vl.commitChanges()

#---------------------------------------------------------------    
# Append attributes
#
vl.startEditing()

col_id = 0
for n in self.dlg.selectionlist:

    col_id += 1

    # Load Values (1 String, 1 Double, Double Array with mesh.head.npoin values)
    aName,aTime,aVALUES = ReadSelafin_OneParaSet(n[0],n[1],mesh)        

    # New Attribute Col
    pr.addAttributes( [QgsField(aName, QVariant.Double) ] )

           #------
           # Add Values to Triangle Features in Values            
           i = 1 
           for TRI in mesh.ELE:

               ep1 = TRI.P1-1
               ep2 = TRI.P2-1
               ep3 = TRI.P3-1
               eleval = (aVALUES[ep1]+aVALUES[ep2]+aVALUES[ep3])/3

               attrs = { i : {col_id : eleval}}
               pr.changeAttributeValues( attrs )
               i += 1       

#---------------------------------------------------------------
# Commit changes
vl.commitChanges()
vl.updateExtents() 
QgsMapLayerRegistry.instance().addMapLayer(vl)
iface.mapCanvas().refresh()

Answer 1

I had a similar problem where I was writing attributes into thousands of features and the writing would get progressively slower and slower as the loop iterated.

I found that if I periodically saved / closed off the edits and then restarted editing (after each row of the array in my case) this made things MUCH faster:

    if currentrow > lastrow:
      lastrow = currentrow
      boxlayer.updateExtents()
      boxlayer.commitChanges()
      boxlayer.startEditing()   

Using this method I was able to change a process that started at 0.02 second execution and increased to 2 second execution after 27k features to maintain a loop time of under 0.04 seconds average.

Answer 2

THe short answer is: I can;t see a way. However, changing attributes on a feature is time consuming at times (I have have a hard time figuring out why it happens). Thus, this should speed up your code a bit:

pr.addAttributes( [QgsField(aName, QVariant.Double) ] )
attrs={}
       #------
       # Add Values to Triangle Features in Values            
       i = 1 
       for TRI in mesh.ELE:

           ep1 = TRI.P1-1
           ep2 = TRI.P2-1
           ep3 = TRI.P3-1
           eleval = (aVALUES[ep1]+aVALUES[ep2]+aVALUES[ep3])/3

           attrs[i]= {col_id : eleval}

           i += 1
pr.changeAttributeValues( attrs )   

Answer 3

OK - I totally forgot about this question that I had previously answered and I currently have a new answer which is at least 10x as fast.

The answer is this: Do not modify the attributes directly.

Collect all of the data you want for all of the attributes into a pandas dataframe. generate a list of features:

for row in dataframe:
  feat = QgsFeature()
  feat.setAttributes(row.values.tolist())
  feat.setGeometry(featgeom)
  feats.append(feat) 

Then delete all the features from your layer and add your new features in:

layer.deleteFeatures(boxlayer.allFeatureIds())
layer_provider.addFeatures(feats)

This will solve your problem (and anyone who comes looking for this post in the future!)