3

Context: I'm writing a dynamic import tool for QGIS (2.2 to 2.5) with python. Datatype: Binary Datasets with environmental simulations from HPC which might contain millions of features and terabytes of attributes.

Current status: I wrote some python that extracts the features and attributes at a certain time step.

  • Loading Data for 100000 features takes < 0.5s via python.
  • Displaying them takes 30s.
  • Data structure: Double precision arrays

Now my question in short: Is it possible to speed up the change of attributes? Can I change 10 attributes for 100 000 features in one step? Rather then doing 1000000 loops, one per attribute?

I know QGIS would be fast enough, because: If I save my imported data as shapefile and reload it, the shapefile reader takes only ~2.5sec for 1 000 0000 attributes.

Here is my current code:

#---------------------------------------------------------------
# Create features
#
#
vl = QgsVectorLayer("polygon", myFile + " | Triangles", "memory")      # New memory layer
pr = vl.dataProvider()                                                 # Dataprovider

# Enter editing mode
vl.startEditing()

#------
# Attribut col
pr.addAttributes( [ QgsField("ID",  QVariant.Int) ] )   

#------
# Paint Triangles
i = 0 
for TRI in mesh.ELE:
    i += 1
    feat = QgsFeature()
    ep1 = TRI.P1-1
    ep2 = TRI.P2-1
    ep3 = TRI.P3-1
   feat.setGeometry(QgsGeometry.fromPolygon([[ \ QgsPoint(mesh.HEAD.X_origin+mesh.x[ep1],mesh.HEAD.Y_origin+mesh.y[ep1]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep2],mesh.HEAD.Y_origin+mesh.y[ep2]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep3],mesh.HEAD.Y_origin+mesh.y[ep3]), \
    QgsPoint(mesh.HEAD.X_origin+mesh.x[ep1],mesh.HEAD.Y_origin+mesh.y[ep1])]]))
    attrdict = [i,]
    feat.setAttributes(attrdict)
    (res, outFeats) = pr.addFeatures( [ feat ] )

vl.commitChanges()

#---------------------------------------------------------------    
# Append attributes
#
vl.startEditing()

col_id = 0
for n in self.dlg.selectionlist:

    col_id += 1

    # Load Values (1 String, 1 Double, Double Array with mesh.head.npoin values)
    aName,aTime,aVALUES = ReadSelafin_OneParaSet(n[0],n[1],mesh)        

    # New Attribute Col
    pr.addAttributes( [QgsField(aName, QVariant.Double) ] )

           #------
           # Add Values to Triangle Features in Values            
           i = 1 
           for TRI in mesh.ELE:

               ep1 = TRI.P1-1
               ep2 = TRI.P2-1
               ep3 = TRI.P3-1
               eleval = (aVALUES[ep1]+aVALUES[ep2]+aVALUES[ep3])/3

               attrs = { i : {col_id : eleval}}
               pr.changeAttributeValues( attrs )
               i += 1       

#---------------------------------------------------------------
# Commit changes
vl.commitChanges()
vl.updateExtents() 
QgsMapLayerRegistry.instance().addMapLayer(vl)
iface.mapCanvas().refresh()
  • I just want to ask this question again! After more than 3 years the problem is still unsolved! – UHM Jan 10 '18 at 9:20
4

I had a similar problem where I was writing attributes into thousands of features and the writing would get progressively slower and slower as the loop iterated.

I found that if I periodically saved / closed off the edits and then restarted editing (after each row of the array in my case) this made things MUCH faster:

    if currentrow > lastrow:
      lastrow = currentrow
      boxlayer.updateExtents()
      boxlayer.commitChanges()
      boxlayer.startEditing()   

Using this method I was able to change a process that started at 0.02 second execution and increased to 2 second execution after 27k features to maintain a loop time of under 0.04 seconds average.

1

THe short answer is: I can;t see a way. However, changing attributes on a feature is time consuming at times (I have have a hard time figuring out why it happens). Thus, this should speed up your code a bit:

pr.addAttributes( [QgsField(aName, QVariant.Double) ] )
attrs={}
       #------
       # Add Values to Triangle Features in Values            
       i = 1 
       for TRI in mesh.ELE:

           ep1 = TRI.P1-1
           ep2 = TRI.P2-1
           ep3 = TRI.P3-1
           eleval = (aVALUES[ep1]+aVALUES[ep2]+aVALUES[ep3])/3

           attrs[i]= {col_id : eleval}

           i += 1
pr.changeAttributeValues( attrs )   
  • 1
    It's a real shame that there seems to be no direct way to replicate the mechanism of the 'calculate' tool in the attribute gui from within a python script. The calculation tool appears instant but the very same calculation carried out a 'feature iterator loop' in python is painfully slow. – Mr Purple Nov 25 '14 at 1:42
  • Thanks for the idea, I tried it, but with limited speed up. Isn't there a way for a direct memory access? I load the same data with python in Paraview which takes 10sec for 600 000 triangles. QGIS takes >1h ... And, not only changing attributes, but also shift painting triangles or shifting geometry. – UHM Nov 25 '14 at 7:16
  • I am not aware of that possibility, and it might be the case that you would have to do it through QGIS' C API. Another issue to work with is the notoriously slow python loops. Maybe playing with different iterators might help your cause. – Pedro Camargo Nov 25 '14 at 22:46
1

OK - I totally forgot about this question that I had previously answered and I currently have a new answer which is at least 10x as fast.

The answer is this: Do not modify the attributes directly.

Collect all of the data you want for all of the attributes into a pandas dataframe. generate a list of features:

for row in dataframe:
  feat = QgsFeature()
  feat.setAttributes(row.values.tolist())
  feat.setGeometry(featgeom)
  feats.append(feat) 

Then delete all the features from your layer and add your new features in:

layer.deleteFeatures(boxlayer.allFeatureIds())
layer_provider.addFeatures(feats)

This will solve your problem (and anyone who comes looking for this post in the future!)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.