4

I have a shapefile with a little over one million features (but it could also be 10+ million).

I need to categorize them by an attribute they have. It is a float value, and the categories are:

1: attrib < 0.1
2: attrib >= 0.1 and < 0.3
3: attrib >= 0.3 and < 0.5
4: attrib >= 0.5

I want to add this category value in a new field.

Easy enough using the field calculator, I thought:

CASE
    WHEN attrib < 0.1 THEN 1
    WHEN attrib >= 0.1 AND attrib < 0.3 THEN 2
    WHEN attrib >= 0.3 AND attrib < 0.5 THEN 3
    WHEN attrib < 0.5 THEN 4
END

into a new field ('category', integer, length 1).

This process is very slow (about a minute per 1000 features).

If I just style the features according to the same rules (graduated, 4 classes), that happens in a manner of seconds and I get a fancy visualization, which I take to mean there has to be a faster way to do this.

So the question: Why is the field calculator so much slower than the graduated style, and what can I do to speed this up? I am not opposed or unfamiliar with pyqgis, so a python/processing script solution is just as fine. I attempted a small script that basically does the same as the field calculator above, but it has the same low performance:

##input_layer=vector
##General Tools=group
##Test=name

from qgis.core import *

layer = processing.getObject(input_layer)
layer.startEditing()
for i, f in enumerate(layer.getFeatures()):
    level = f['attrib']
    if level < 0.1:
        f['cat'] = 1
    elif level >= 0.1 and level < 0.3:
        f['cat'] = 2
    elif level >= 0.3 and level < 0.5:
        f['cat'] = 3
    elif level >= 0.5:
        f['cat'] = 4
    layer.updateFeature(f)
    if i % 1000 == 0: # Only added this to track how slow the script is
        print i
        progress.setText('Feature ' + str(i))
layer.commitChanges()

Running QGIS 2.18.15.

3

@NyallDawson, one of the core developers of QGIS, posted a blog: "Speeding up your PyQGIS scripts" where he advised on using two methods when iterating through large datasets (described in "Trap #3: Only request values you need"):


So you could try using the following where hopefully it should take slightly less time:

##input_layer=vector
##General Tools=group
##Test=name

from qgis.core import QgsFeatureRequest

layer = processing.getObject(input_layer)
request = QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(['attrib'], layer.fields())
layer.startEditing()
for i, f in enumerate(layer.getFeatures(request)):
    level = f['attrib']
    if level < 0.1:
        f['cat'] = 1
    elif level >= 0.1 and level < 0.3:
        f['cat'] = 2
    elif level >= 0.3 and level < 0.5:
        f['cat'] = 3
    elif level >= 0.5:
        f['cat'] = 4
    layer.updateFeature(f)
layer.commitChanges()

Note: I have used these methods on layers with thousands of features which worked very well. But not on those with a million features, I would opt to use PostGIS instead.

  • 1
    Thanks! I did a test run, stopping time.time() difference for both approaches and the first 10k samples. My approach took 40 seconds, yours cut it to 20. still feels way slower than it should need to be. – Senshi Dec 21 '17 at 14:28
  • 1
    @Senshi - Thanks for testing. I agree that the method I posted may not be the most efficient but I think the field calculator itself is not designed to work in a database manner. Especially with the number of features you are using, you are probably much better off importing the shapefile into PostGIS and use the powerful database capabilities it has. I wouldn't be surprised if it did the exact same thing in seconds :) – Joseph Dec 21 '17 at 14:45
  • 2
    @Senshi - QGIS is fast when reading values but not as fast when writing and updating values, especially to disk. Hopefully someone with knowledge of the source code could enlighten us =) – Joseph Dec 21 '17 at 15:04
  • 1
    With a shapefile you may have to write the whole thing out to a new file when adding an attribute – Ian Turton Dec 26 '17 at 17:43
  • 1
    Another update: Going the PostGIS route: 1 074 547 (just over 1mio) features updated in 21 sec. Nice: The query is exactly the same as for the field calculator. Lesson learned: QGIS can read and juggle lots of data fast, but it's bad at writing it fast (at least in shape DBFs, which one can hardly blame it for). – Senshi Jan 2 '18 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.