6

I am currently working on an add-on that relies on the user choosing fields of tables (that can be from shapefiles, csv or dbf) that are open in QGIS and then making computations with such data.

Right now I am looping through all the features and collecting the data using the standard cookbook techniques to loop through features and to read attributes, but it is taking too long. Some tables have about 25,000,000 features, and the actual data loading takes much longer than the computing. Is there any shortcut? Can I load whole vectors at once or some other similar thing?

It occurs to me that I can just get the file information and go directly to the database in disk and load it from there, but I REALLY want to avoid it.

It seems that the question was not very clear, so here goes what I have now.

all_matrix=[]
for f in matrix.getFeatures():
    a=feat.attributes()[origin]
    b=feat.attributes()[destination]
    c=feat.attributes()[flow]
    all_matrix.append([a,b,c])
 matrix=np.array(all_matrix)
3
  • 1
    use ogr library or shapely library which integrate numpy functions Commented Sep 22, 2014 at 17:23
  • What format is your data in? Shapefile? Postgres? Commented Mar 19, 2015 at 10:31
  • Any data format that QGIS would accept, probably it would be csv or dbf most of the times. Commented Mar 20, 2015 at 14:33

2 Answers 2

3

What about the QgsFeatureRequest.NoGeometry flag?

all_matrix=[]
request = QgsFeatureRequest()
request.setFlags(QgsFeatureRequest.NoGeometry)
for feat in matrix.getFeatures(request):
    a=feat.attributes()[origin]
    b=feat.attributes()[destination]
    c=feat.attributes()[flow]
    all_matrix.append([a,b,c])
 matrix=np.array(all_matrix)
2
  • 1
    OK I just saw how old this is... meh.
    – Mr Purple
    Commented Jul 8, 2015 at 2:47
  • You may get even better performance by generating the matrix in the layout the fields are received. all_matrix=[f.attributes() for f in l.getFeatures(request)]. If not all attributes are required, you can further reduce network overhead by using QgsFeatureRequest.setSubsetOfAttributes Commented Jan 28, 2016 at 11:17
1

You can retrieve the selected features with

vl = iface.acitveLayer()
selectedFeatures = vl.selectedFeatures()

In case you have many features, may be you can use selectedFeaturesIds() and then query and iterate over these to avoid copying all the features into the memory.

1
  • Anand, this is what I am doing, and it is extremely slow. Please see the update in the question. Commented Sep 22, 2014 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.