3

QGIS tools change special characters (õ, à) of attributes into unreadable characters.

Example:

# load 1 wfs feature
uri = "http://geo.vliz.be/geoserver/MarineRegions/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=MarineRegions:eez&Filter=%3Cogc:Filter%20xmlns:ogc=%22http://www.opengis.net/ogc%22%3E%3Cogc:PropertyIsLike%20matchCase=%22false%22%20wildCard=%22*%22%20singleChar=%22.%22%20escapeChar=%22!%22%3E%3Cogc:PropertyName%3Egeoname%3C/ogc:PropertyName%3E%3Cogc:Literal%3E*Gomera*%3C/ogc:Literal%3E%3C/ogc:PropertyIsLike%3E%3C/ogc:Filter%3E"
vlayer = QgsVectorLayer(uri, "test", "WFS")

# save as shapefile with UTF-8 encoding
QgsVectorFileWriter.writeAsVectorFormat(layer = vlayer, fileName = 'H:/Gomera.shp', fileEncoding = 'utf-8', driverName = 'ESRI Shapefile')

# run qgis tool    
import processing
processing.run("native:fixgeometries", {'INPUT':'H:/Gomera.shp','OUTPUT':'H:/Gomera_fix.shp'})

In the output shapefile, the special characters have been changed into unreadables (compare left to right attribute tables):

Qgis changes special characters

The input shapefile and output shapefiles are both UTF-8 encoded. This happens with several tools, for example with fixgeometries, buffer or grass7:v.buffer, both in the python console as well as in the GUI.

How can I solve this problem? This seems something related to internal QGIS tools rather than problems with reading special characters as input as discussed here:

I'm running QGIS 3.0 on Windows.

|improve this question
  • 1
    The default encoding of shapefiles is ISO8859_1 gdal.org/drv_shapefile.html. Does your script create also .cpg file that defines the encoding? It may happen that you have written UTF-8 but QGIS believes that it is ISO 8859-1. Have you considered to use GeoPackage as an outputformat? – user30184 Apr 17 '18 at 15:53
  • Just saving the the shapefile creates a .cpg file with 'utf-8', but the out shapefiles from qgis tools indeed lack a .cpg file. Adding this file manually doesn't solve the problem. The GeoPackage alternative works, but this doesn't solve the shapefile issue? – Lennert Apr 18 '18 at 7:19
  • Perhaps the tools do not write out UTF-8. What if you try with a .cpg file that contains string OEM 1252. If you are willing to use hex editor you could also check the values of byte 30 (offset 29) from the beginning of the working and not working .dbf files. – user30184 Apr 18 '18 at 8:05
  • OEM 1252 in .cpg gives the same (unreadable) result. byte 30 of the .dbf files is both 00, this is part of the header. It seems that the output .dbf has another encoding though, when I change it to 'Latin-1' as mentioned here: gis.stackexchange.com/a/255536/44980 the special characters show up fine. – Lennert Apr 18 '18 at 10:19
3

Use:

orgEncoding=QgsSettings().value('/Processing/encoding') # save setting
QgsSettings().setValue('/Processing/encoding', 'utf-8') # set uft8

# run qgis tool    
import processing
processing.run("native:fixgeometries", {'INPUT':'H:/Gomera.shp','OUTPUT':'H:/Gomera_fix.shp
QgsSettings().setValue('/Processing/encoding', orgEncoding) # set saved value

For set the value manuellenter image description here

|improve this answer
  • Hi Mike, this works! It seems that my processing encoding is empty, so would this have implications if I just leave it in utf-8? In my processing options, the encoding setting has now the value 'Invalid'. Do you know why the default setting could be set to 'Invalid' in my case? – Lennert Apr 18 '18 at 10:34
  • The value is a saved setting (registry in Windows) of the Python plug-in. You can also set the value via dialog box. (I added a picture above) – Mike Apr 18 '18 at 12:39
1

I want to highlight some issues that I found while looking into my problem here:

  1. There is no official way to define the encoding of the attributes (.dfb) of a shapefile, storing the encoding in a .cpg file is not accepted/read by all programs. See Which character encoding is used by the dbf file in shapefiles.

  2. In QGIS, shapefile encoding problems might arise from two problems:

    • The QGIS processing 'encoding' parameter. You can change this setting as in Mike's answer:

      orgEncoding=QgsSettings().value('/Processing/encoding') # save setting
QgsSettings().setValue('/Processing/encoding', 'utf-8') # set uft8
# revert setting:
QgsSettings().setValue('/Processing/encoding', orgEncoding) # set saved value

    • The 'Ignore shapefile encoding declaration'. By default, the shapefile encoding mentioned in .cpg files is ignored in QGIS. If you want QGIS to detect the encoding and converse it 'on the fly' to UTF-8, uncheck this option.

      Options > Data Sources > uncheck 'Ignore shapefile encoding declaration'.

      More info in the QGIS issue When opening Shapefile the .cpg file is ignored

  3. There seems to be whole Geo IT groups urging for the replacement of Shapefiles (http://switchfromshapefile.org), so maybe it's indeed better to shift from the Shapefile to the OGC GeoPackage format.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.