QGIS tools change special characters to unreadables

Question

QGIS tools change special characters (õ, à) of attributes into unreadable characters.

Example:

# load 1 wfs feature
uri = "http://geo.vliz.be/geoserver/MarineRegions/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=MarineRegions:eez&Filter=%3Cogc:Filter%20xmlns:ogc=%22http://www.opengis.net/ogc%22%3E%3Cogc:PropertyIsLike%20matchCase=%22false%22%20wildCard=%22*%22%20singleChar=%22.%22%20escapeChar=%22!%22%3E%3Cogc:PropertyName%3Egeoname%3C/ogc:PropertyName%3E%3Cogc:Literal%3E*Gomera*%3C/ogc:Literal%3E%3C/ogc:PropertyIsLike%3E%3C/ogc:Filter%3E"
vlayer = QgsVectorLayer(uri, "test", "WFS")

# save as shapefile with UTF-8 encoding
QgsVectorFileWriter.writeAsVectorFormat(layer = vlayer, fileName = 'H:/Gomera.shp', fileEncoding = 'utf-8', driverName = 'ESRI Shapefile')

# run qgis tool    
import processing
processing.run("native:fixgeometries", {'INPUT':'H:/Gomera.shp','OUTPUT':'H:/Gomera_fix.shp'})

In the output shapefile, the special characters have been changed into unreadables (compare left to right attribute tables):

The input shapefile and output shapefiles are both UTF-8 encoded. This happens with several tools, for example with fixgeometries, buffer or grass7:v.buffer, both in the python console as well as in the GUI.

How can I solve this problem? This seems something related to internal QGIS tools rather than problems with reading special characters as input as discussed here:

• QGIS Processing algorithm cannot deal with special characters in input
• ArcPy Field Calculation with special characters

I'm running QGIS 3.0 on Windows.

Answer 1

Use:

orgEncoding=QgsSettings().value('/Processing/encoding') # save setting
QgsSettings().setValue('/Processing/encoding', 'utf-8') # set uft8

# run qgis tool    
import processing
processing.run("native:fixgeometries", {'INPUT':'H:/Gomera.shp','OUTPUT':'H:/Gomera_fix.shp
QgsSettings().setValue('/Processing/encoding', orgEncoding) # set saved value

For set the value manuell

Answer 2

I want to highlight some issues that I found while looking into my problem here:

1. There is no official way to define the encoding of the attributes (.dfb) of a shapefile, storing the encoding in a .cpg file is not accepted/read by all programs. See Which character encoding is used by the dbf file in shapefiles.

2. In QGIS, shapefile encoding problems might arise from two problems:

   • The QGIS processing 'encoding' parameter. You can change this setting as in Mike's answer:

     orgEncoding=QgsSettings().value('/Processing/encoding') # save setting
     QgsSettings().setValue('/Processing/encoding', 'utf-8') # set uft8
     # revert setting:
     QgsSettings().setValue('/Processing/encoding', orgEncoding) # set saved value
     

   • The 'Ignore shapefile encoding declaration'. By default, the shapefile encoding mentioned in .cpg files is ignored in QGIS. If you want QGIS to detect the encoding and converse it 'on the fly' to UTF-8, uncheck this option.

     Options > Data Sources > uncheck 'Ignore shapefile encoding declaration'.

     More info in the QGIS issue When opening Shapefile the .cpg file is ignored

3. There seems to be whole Geo IT groups urging for the replacement of Shapefiles (http://switchfromshapefile.org), so maybe it's indeed better to shift from the Shapefile to the OGC GeoPackage format.