Performance-limits of QGIS expressions

Question

I use quite often QGIS expressions and I am wondering how performant they are compared with other possibilities. In many cases, the same problem can be resolved using different approaches like using either expressions, or a combination of GUI tools, or a query in a virtual layer, or a Python script (and others, like PostGIS).

Apart from the situation when certain goals cannot be realized by using expressions, I'm interested in the limitations regarding performance (an example below, update 2). So the question is: are there any clear criteria when expressions hit a performance limitation and turn to be less performant than Virtual layers or Python scripts?

Regarding some more complex expressions, I realized that they can quite take some time to update when used for "realtime" rendering as in these expamples here:

Re-locating points randomly at certain distance in QGIS

Create road network from points

Update 2: One limitation in the kind that I am asking for is mentioned by Ujaval Gandhi in a recent tutorial:

aggregate expressions are not yet optimized with spatial indexing

see QGIS with Ujaval Gandhi, at 1:02:21 to 1:02:53, https://youtu.be/IraiAF4s5Tg?t=3741

Update 1, to be more precise: I am primarily interested in the performance of the algorithms that run in the background when using expressions (compared to virtual layers, python etc.). Of course, using expressions for symbol layers with geometry generator is nice, but with a a lot of features and complex calculations, this dynamic approach of permanent redrawing and calculating the map canvas can be slow. But this is not primarily a limitation of expressions, but of their use for "on the fly" creation of geometries and affects virtual layers as well. Using expressions with the geometry by expression tool does not have this limitation - however the question of performance is still relevant there.

---

See here for some information about Performance boost of QGIS Expressions Engine, which, however, does not directly answer the question here.

Answer 1

This answer is written in the context of the use-case of computational use of expressions, i.e. rather than in symbol display or geometry generator, as per the updated question.

First, any use of an expression carries a slight performance hit due to parsing and tokenizing the expression string (referred to in the performance optimization link as QgsExpression.prepare(). This hit might be meaningful if you use simple expressions many, many times, but fades away if you are using expressions which are themselves computationally intensive. The field calculator appropriately parses and tokenizes once, then loops across records. However, beware of expression use in instances where you loop on the outside, and expressions are being parsed and tokenized inside in each iteration.

I also believe but have not verified that caching (discussed in the performance optimization link) and the careful tagging into static/non-static is not remembered across separate expression evaluation invocation, so if you parse/tokenize inside a loop or iteration, you won't get the benefit of caching across iterations.

Finally, as you mention, aggregation functions do not use spatial indexing. And any use of $geometry (or similar) will mark that node and all its parents as non-static.

Therefore, an attempt at rules of thumb:

1. Try to use expressions where QGIS runs the looping, rather than push a new expression evaluation instance inside a loop.

2. If your expression contains multiple uses of the same fragment, especially if that fragment refers to non-static elements such as $geometry, use a with_variable() construct to calculate it only once per expression.

3. If a sub-expression includes looking into other records, especially aggregation, that is when you are most likely to see benefit from "pre-calculating" those pieces as a (possibly temporary) new field or virtual field first.

4. The previous point has a countervailing effect if you would be precalculating in a large, especially remotely-stored, layer where your final results only depend on a local region. In this case, consider exporting a neighbourhood of your local region as a locally saved (even temporary) layer and pre-calculating only there.

5. Be careful of extensive use of aggregate over large layers; may be worthwhile to rewrite using processing instead to benefit from indexing.

6. If you find yourself optimizing your expression with multiple nested with_variables, that's evidence using a processing sequence or writing a PyQGIS script may be more performant. (Whether it's worth your time to do so is a different question.)