0

I ambuilding a python plugin for QGIS. As part of the plugin, I do: iface.addVectorLayer(geoj, self.selectedLayer, 'ogr')

This is basically adding a layer to the project, with geoj: a geojson string, self.selectedLayer: the name I want to give it, and 'ogr': the driver.

I get a layer in my project with most field lengths set to zero: enter image description here

This is a problem, because when I then try to export the loaded layer as a shapefile or something, I get the following error: *Export to vector file failed. Error: Feature write errors: Error converting value ((1:0)) for attribute field photos_captions: String of length 5 exceeds maximum field length (1) *

And a complete QGIS Crash!

Do I need to specify some more parameters to be able to load the geojson in my project as a valid layer with correctly defined fields?

Here are the relevant sections of code:

 def getGeoJsonFromSelectedLayer(self):

        if self.selectedLayer:
            url = "https://api.fulcrumapp.com/api/v2/query"

            tableSelector = f"SELECT * FROM \"{self.selectedLayer}\""
            querystring = {"q": tableSelector, "format": "geojson", "headers": "false",
                           "metadata": "false", "arrays": "false", "page": "1", "per_page": "20000"}

            headers = {
                "Accept": "application/json",
                "X-ApiToken": self.API_TOKEN
            }

            response = requests.request(
                "GET", url, headers=headers, params=querystring)

            # For Debugging:
            iface.messageBar().pushMessage(response.text)

            self.createLayerFromGeojson(response.text)

        else:
            iface.messageBar().pushMessage("No Layer Selected")

    def createLayerFromGeojson(self, geoj):
        # if there are features in the list
        if len(geoj) > 0:
                       
            # add the layer to the list of layers
            iface.addVectorLayer(geoj, self.selectedLayer, 'ogr')

        else:
            iface.messageBar().pushMessage("No features found in the geoJSON")
4
  • What is the type of self.selectedLayer ?
    – JULESG
    Mar 15 at 7:38
  • its just a string
    – staf
    Mar 15 at 7:58
  • It looks like there is an issue with the type of field "IntegerList" in Qgis : gis.stackexchange.com/questions/341443/… In your geojson the value of this field must be : "photos_captions":[0] but in Qgis it looks like this "photos_captions" : (1:0). It means it's a list of length 1 with 0 as value. You need to convert this value (1:0) to 0. If you always have one value you can just delete '[' and ']' in your geojson.
    – JULESG
    Mar 15 at 8:24
  • MORE INFO: The error does not present with QGIS 3.10.14 Long Term Release. The error DOES present with QGIS 3.16.3
    – staf
    Mar 16 at 2:35
1

To answer your question and to add some code to my comment :

I writed a snippet of code based on this answer :

import re
    
geoj_layer = QgsVectorLayer(geoj, '', 'ogr')
    refactor = []
field_error = []
for field in geoj_layer.fields(): # creation of a list with the field and field type of the layer.
    refactor_field = {}
    refactor_field['expression']=str("\"")+field.name()+str("\"")
    refactor_field['name']=field.name()
    refactor_field['precision']=field.precision()
    if field.typeName() == "IntegerList": # if the field as a not well handled type
        field_error.append(field.name()[:10])
        refactor_field['length']=0
        refactor_field['type']=10
    else :
        refactor_field['type']=field.type()
    refactor.append(refactor_field)

output_path = path + 'new_layer.shp'
# I'm using refactorfield algorithm from Qgis to create a new layer with some modification on specified fields.
processing.run("qgis:refactorfields", {'INPUT':geoj,'FIELDS_MAPPING':refactor,'OUTPUT':output_path})
iface.addVectorLayer(output_path,'', 'ogr')

# Modification of the value for a better value
geoj_layer = QgsVectorLayer(output_path, '', 'ogr')
for field_name in field_error:
    for f in geoj_layer.getFeatures() :
        attrs= f.attributes()
        value = attrs[geoj_layer.fields().indexFromName(field_name)]
        string = re.split(',|:|\)', value)
        new_value = ''
        for elem in string[1:-1] :
            if elem == string[1:-1][-1] :
                new_value = new_value +str(elem)
            else :
                new_value = new_value +str(elem) + ", "
        geoj_layer.startEditing()
        geoj_layer.changeAttributeValue(f.id(), geoj_layer.fields().indexFromName(field_name), new_value )
        geoj_layer.commitChanges()
        geoj_layer.triggerRepaint()

This code will create a new shapefile with a field "photos_cap" as a String. Shapefile only accept 10 characters for field name.You'll get a value of 0 instead of (1:0).

Edit after the author's comment

You can still use refactorfield on a geojson and create another geojson with the column and value you want.

Example :

This is my geojson with the IntegerList :
enter image description here

I can obtain this kind of result in a temporary layer : enter image description here

Instead of calling the geojson from your api with iface.addVectorLayer you can call it with QgsVectorLayer so it doesn't appear in the canvas. You can then use refactorfield to change the fields' type of your layer and save the result in a temporary layer (I use tempfile to save the layer in a temporary directory).

With this code, only one layer appear in the canvas, a temporary layer without IntegerList :

import re, tempfile
tf = tempfile.TemporaryDirectory()

def format_to_string(value):
    if isinstance(value, str) :
        string = re.split(',|:|\)', value)
        new_value = ''
        for elem in string[1:-1] :
            if elem == string[1:-1][-1] :
                new_value = new_value +str(elem)
            else :
                new_value = new_value +str(elem) + ", "
    if isinstance(value, list) :
        string = ""
        for elem in value:
            string += elem
            if elem != value[-1] :
                string = string + ', '
        new_value = string
    return (new_value)

path = tf.name + "/"

geoj = "/home/Documents/geojson/champ_listint.geojson"

geoj_layer = QgsVectorLayer(geoj, '', 'ogr')
refactor = []
field_error = []
for field in geoj_layer.fields(): # creation of a list with the field and field type of the layer.
    refactor_field = {}
    refactor_field['expression']=str("\"")+field.name()+str("\"")
    refactor_field['name']=field.name()
    refactor_field['precision']=field.precision()
    if field.typeName() == "IntegerList": # if the field as a not well handled type
        field_error.append(field.name())
        refactor_field['length']=0
        refactor_field['type']=10
    elif field.typeName() == "StringList": # if the field as a not well handled type
        field_error.append(field.name())
        refactor_field['length']=0
        refactor_field['type']=10
    else :
        refactor_field['type']=field.type()
    refactor.append(refactor_field)

output_path = path + 'new_layer.geojson'
# I'm using refactorfield algorithm from Qgis to create a new layer with some modification on specified fields.
processing.run("qgis:refactorfields", {'INPUT':geoj,'FIELDS_MAPPING':refactor,'OUTPUT':output_path})
iface.addVectorLayer(output_path,'', 'ogr')

# Modification of the value for a better value
geoj_layer_2 = QgsVectorLayer(output_path, '', 'ogr')
for field_name in field_error:
    for f in geoj_layer.getFeatures() :
        attrs= f.attributes()
        value = attrs[geoj_layer.fields().indexFromName(field_name)]
        new_value = format_to_string(value)
        geoj_layer_2.startEditing()
        geoj_layer_2.changeAttributeValue(f.id(), geoj_layer.fields().indexFromName(field_name), new_value )
        geoj_layer_2.commitChanges()
        geoj_layer_2.triggerRepaint()

enter image description here

11
  • Thanks for your detailed answer, but it doesnt completely resolve the issue. My plugin loads data in from an API. The purpose of the plugin is for users to be able to pull in tables from a closed web based database, and then use these as a normal layer in QGIS. So I need to modify the field on the way in... So that by the time it is loaded as an in memory layer in QGIS it is stable, and users can use their normal 'export' methods on it. Any ideas?
    – staf
    Mar 16 at 2:49
  • So is there any way to read in geojson data into a QGIS in-memory layer that has integerList type fields, that is stable? I appreciate that I could do it using an intermediary step of writing a shapefile, but it will add unwanted complexity and field name truncation issues
    – staf
    Mar 16 at 3:07
  • Instead of calling the layer from the API on Qgis with iface.addVectorLayer, you can call it with QgsVectorLayer so it doesn't appear then proceed to a change with the refactor field, save the layer in a temporary directory and add it to the canvas so the user can use it like a normal layer. I edit my answer.
    – JULESG
    Mar 16 at 13:34
  • Hi JULESG I dont think I can accept this as the answer, as I need a way to handle ALL poorly handled fields that could possibly be in the geojson. Also, this code does not work if the values are empty for a row. I get errors like: Error converting value () for attribute field photos: Could not convert value "" to target type
    – staf
    Mar 22 at 7:19
  • 1
    Hi JULESG, thanks again. I get this error with your solution: CRITICAL Feature could not be written to C:\Users\staff\AppData\Local\Temp\tmpyg9xtn3o/new_layer.geojson: Error converting value () for attribute field photos: Could not convert value "" to target type . I really appreciate your time on this, I will try and handle this error.
    – staf
    Mar 23 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.