I am currently working on an add-on that relies on the user choosing fields of tables (that can be from shapefiles, csv or dbf) that are open in QGIS and then making computations with such data.
Right now I am looping through all the features and collecting the data using the standard cookbook techniques to loop through features and to read attributes, but it is taking too long. Some tables have about 25,000,000 features, and the actual data loading takes much longer than the computing. Is there any shortcut? Can I load whole vectors at once or some other similar thing?
It occurs to me that I can just get the file information and go directly to the database in disk and load it from there, but I REALLY want to avoid it.
It seems that the question was not very clear, so here goes what I have now.
all_matrix=[]
for f in matrix.getFeatures():
a=feat.attributes()[origin]
b=feat.attributes()[destination]
c=feat.attributes()[flow]
all_matrix.append([a,b,c])
matrix=np.array(all_matrix)