PyQGIS: Inline function to replace NULL values with 0 in all fields (not coalesce)?

Question

Now I know there are several ways to tackle this problem but so far none of the following worked for me. I got a huge attribute table out of various spatial joins so pretty much all of the fields have NULL values in some rows. This table will be processed and new fields are going to be calculated programatically so the solution involving the coalesce function is out of the race because the expressions for the following calculations are complex enough and I don't want to do all that typing. So here are my failed attempts of fiddeling together a function that i can use inline:

def replaceNulls(layer):
   context = QgsExpressionContext()
   context.appendScopes(QgsExpressionContextUtils.globalProjectLayerScopes(layer))
   with edit(layer):
      for field in layer.fields().names():
          for feat in layer.getFeatures():
              expr = QgsExpression('if({f} is null,0,{f})'.format(f=field))
              feat[field] = expr.evaluate(context)
              layer.updateFeature(feat)

Using an expression from this answer replaces ALL the fields with 0 and some with field - independent of their field type.

def replaceNulls(layer):
   context = QgsExpressionContext()
   context.appendScopes(QgsExpressionContextUtils.globalProjectLayerScopes(layer))
   with edit(layer):
      for field in layer.fields().names():
          for feat in layer.getFeatures():
              expr = QgsExpression('case when {f} is null then 0 else {f} end'.format(f=field))
              feat[field] = expr.evaluate(context)
              layer.updateFeature(feat)

This function replaces all values that are NOT NULL with 0 for some reason all though it is said here that:

The expression syntax is exactly the same as in the field calculator GUI. So you can test them in the GIU first, before copying them into your PyQGIS script.

If that is the case what am I missing out on? The same source also mentions a more pythonic way but how to if feat[field] is None: when the NULL value is still a QVariant?

TL;DR: replace(NULL,0). How?

Answer 1

one way I found to solve the issue of replacing the NULL values with zeros or similar values is the following:

1. I define a function that maps the NULL value to some other value of choice, depending on the QVariant type:

def map_null_to_zero(qvariant_type):
    if qvariant_type == QVariant.Invalid or qvariant_type >= QVariant.Int and qvariant_type <= QVariant.Double:
        return 0
    elif qvariant_type == QVariant.String or qvariant_type == QVariant.Char:
        return "0"
    elif qvariant_type == QVariant.Bool:
        return False
    else:
        raise ValueError("Unexpected Data type {}".format(qvariant_type))

2. Then, I define another function that, given an input list of attributes and attributes' types, it returns a modified list of attributes without NULL:

def modify_null_attr_to_zero(li_attr=None, li_attr_type=None):
    assert len(li_attr) == len(li_attr_type), "The input list have different lengths"
    return [attr if attr != NULL else map_null_to_zero(type) for attr, type in zip(li_attr, li_attr_type)]

Note that null attributes are identified using the comparison operator (==) with the NULL type, which is a special type of the QVariant:

type(NULL)
<class 'PyQt5.QtCore.QVariant'>

Some more information can be found in the Qt documentation.

3. Finally, I can apply the changes to the required layer:

def replace_nulls(layer):
    li_attr_types = [a.type() for a in layer.fields()]
    with edit(layer):
        for f in layer.getFeatures():
            f.setAttributes(modify_null_attr_to_zero(f.attributes(), li_attr_types))
            if not layer.updateFeature(f):
                print("Error in updating feature {}".format(f.id()))

I think this "modular" solution gives you the flexibility of deciding how to convert NULL based on the type of the attribute. I tested it on the open dataset populated_places from the natural earth. Here is how the original attribute table looks like: And here is how it looks after the execution of the replace_nulls function:

I hope this helps.

Answer 2

So despite the very elaborate answer by fastest which for some reason didn't work for QVariant.Integer and QVariant.Double I ended up using a simpler solution. It might be a bit hacky but it's a classical case of "works-for-me":

def replace_nulls(layer):
    with edit(layer):
        for feat in layer.getFeatures():
            for field in feat.fields().names():
                if feat[field] == None:
                    feat[field] = '0'
            layer.updateFeature(feat)