1

I have a Python dictionary with around 450,000 keys and respective values. I want to join this dictionary to an attribute table of a vector dataset with roughly the same amount of features.

First I created a new field where the values are to be stored and I saved its index in a variable. I also saved the index of the field through which I want to join the values of the keys.

layer_prov = layer.dataProvider()
layer_prov.addAttributes([QgsField("new_field", QVariant.Double)])
layer.updateFields()
new_field_idx = layer.fields().indexFromName('new_field')
join_field_idx = layer.fields().indexFromName('join_field')

layeris the vector dataset that I want to join the information from the dictionary to. dict is the dictionary.

for key, value in dict.items():
layer.startEditing()
for feature in layer.getFeatures():
    attrs = feature.attributes()
    join_value = (attrs[join_field_idx])
    join_value = str(join_value)
    if join_value == key:
        feature[new_field_idx]=value
layer.commitChanges()

Is there any way of doing this faster? For every entry in the dictionary this model needs to go through all the features of the vector dataset (450,000 x 450,000 = 202,500,000,000‬ comparisons).

  • Could be wrong but for a quite massive dataset like this, I would report the join at a database level instead of trying to loop at QGIS level. Much more efficient this way IMHO (PS: I don't know your case so there is maybe a reason you made this choice) – ThomasG77 Jun 11 at 15:01
  • You're right. I created the dictionary previously through various operations but for the Join I could use a database. Is there a way of loading a geopackage into a PostgreSQL database (PostgreSQL 12.2 and PostGIS 3.0.1) with Python? I managed to do so for the dictionary but got stuck with the geopackage. Also I found another solution that works much faster than the previous Loop (see accepted answer). – hdi Jun 16 at 12:56
0

Instead of comparing every feature with the keys in the dictionary, I saved the whole dictionary as a csv and simply used the QGIS algorithm Join Attribute Table (probably some SQL plays its part here as well!?) to join the csv to the vector.

#create csv from dictionary
os.chdir(Output_Ordner)
header_list = ["keys", "values"]
habitat_vkp_sum_csv = open("key_value_csv.csv", "w")
writer = csv.writer(key_value_csv, delimiter=',', quotechar=',')
writer.writerow(header_list)
for key, value in dict.items():
    writer.writerow([key, value])
key_value_csv.close()

#perform table join between vector data and csv table
processing.run("native:joinattributestable", {
    'INPUT': layer,
    'FIELD': 'keys', 'INPUT_2': Output_Ordner+'\\'+'key_value_csv.csv', 'FIELD_2': 'keys',
    'FIELDS_TO_COPY': ['values'], 'METHOD': 1, 'DISCARD_NONMATCHING': False, 'PREFIX': '',
    'OUTPUT': Output_Ordner+'\\'+'result'})
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.