2

This is a spin-off of another question I asked previously, but it has grown in scope and deserves its own thread.

While iterating over a vector layer (shapefile) I included the following if-statement with a breakpoint to check for none NULL values. I started noticing that it was finding values even with an empty field "class" (i.e. all values should be NULL)

if feature.attributes()[fieldIdx["class"]] != None:
    log.write("We got one!!")

Most of the time this behaves as expected (i.e. it is skipped), but occasionally this is triggered. While in the interactive debugging console I poked around a bit.

feature.attributes()
>>> [NULL, NULL, NULL, NULL, 388.0, 9.0, 8.0, 8.0, 1.0, 1, NULL, NULL, u'******************************************************************************************']

This is strange on many levels. The first 4 fields should be defined, but they are NULL. In particular the first field is the feature ID. The order is not mixed. The values are just... wrong. fieldNameIndex shows the expected index.

feature.fieldNameIndex("class")
>>> 9
feature.fieldNameIndex("ID")
>>> 0

Furthermore, if I try grabbing the feature again. I get the correct values with the NULL in the right place.

prov = layer.dataProvider()
f = prov.getFeatures(QgsFeatureRequest(4089)).next()
f.attributes()
>>> [4089, 87758.61373109999, 88258.61373109999, 67765.346125, 68265.346125, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL]

Is my file corrupt? Is this a bug in QgsFeatureIterator?

EDIT: I still can't reproduce the behavior with the most minimal code, even using the original file.

layer = iface.mapCanvas().layer(0)
prov = layer.dataProvider()
layer.startEditing()
if prov.fieldNameIndex("class") == -1:
    prov.addAttributes([QgsField("class",QVariant.Int)])
if prov.fieldNameIndex("testField") == -1:
    prov.addAttributes([QgsField("testField",QVariant.Double)])

layer.updateFields()
layer.commitChanges()
idx = prov.fieldNameIndex("class")
testIdx = prov.fieldNameIndex("testField")

featureator = layer.getFeatures()
for feature in featureator:
    ID = feature.id()

    if feature.attributes()[idx] != None:
        print ID
        print feature.attributes()
        message = "feature {0} has a non-NULL class and the following attributes: {1}".format(ID,feature.attributes())
        QgsMessageLog.logMessage(message,  "Debug",  0)
        break

    # perhaps frequent editing/committing in loop can create problems?
    layer.startEditing()
    layer.changeAttributeValue(ID, testIdx, ID/3)
    layer.commitChanges()
  • what does the attribute table show? – Nathan W Jan 27 '16 at 9:14
  • It shows the values in the second attributes() call. [4089, 87758.61373109999, 88258.61373109999, 67765.346125, 68265.346125, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL] EDIT: The one where I explicitly grab the feature in question. – Kris Jan 27 '16 at 9:32
  • 1
    Upload the shapefile somewhere if you can please for testing. – Nathan W Jan 27 '16 at 10:35
  • @NathanW uploaded to: link – Kris Jan 27 '16 at 10:58
  • I have worked around the problem by explicitly looping through all the features, however this experience has undermined my faith in QgsFeatureIterator and going forward I'm not sure I can use it or recommend its use. Unfortunately, I do not have the time to debug this at the moment. – Kris Feb 16 '16 at 12:35
1

I'm not sure what low level I/O conflict was taking place, but the shapefile was in a folder that was set to sync with a cloud storage provider. I was writing to file at least every 30 seconds and the sync utility was dutifully attempting to keep up. It seems this would occasionally cause QgsFeatureIterator to grab a bad chunk of data and, after looking at my log, also skip some features. After turning off the cloud sync the skipping and the bad data issue seems to have stopped.

UPDATE: This seems to have only decreased the likelihood of it happening, but the error still persists. This is still an open question. Perhaps I should decrease the frequency of the commit operations...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.