Encoding issues (UTF-8 to latin1) in the production of shapefiles using QGIS API

Question

We are facing encoding problems with shapefiles trying to convert UTF-8 encoding to latin1/ISO8859-1 using QGIS API.

In QGIS desktop over the layer added to the TOC doing "save as" and specifying the new encoding (latin1) produces a new shapefile with the encoding we expect, the dbf file is good since the text in attribute table is correct; although since we need a way to automate it over all the shapefiles, we have tried using API - quis.core and PyQt4.QtGui - for the processing, and it creates a new shapefile theoretically in latin1 encoding (properties says latin1 and .cpg file also), but when we inspected it, the text/characters are incorrect. We have used directly the QgsVectorFileWriter class, and seems it does not work properly over the dbf file..!? Using QGIS 2.12 and python 2.7 (x86).

Any idea how to solve the weird behavior/issue?

Main code snippet:

app = QApplication([], True)
QgsApplication.setPrefixPath(r"C:\Program Files (x86)\QGIS Lyon\apps\qgis\\", True)
QgsApplication.initQgis()

layer = QgsVectorLayer(<path_to_input_shp>, "lyr_from_shp", "ogr")
if not layer.isValid():
    print ("Layer {} did not load".format(layer.name()))

writer = QgsVectorFileWriter.writeAsVectorFormat(layer, <path_to_output_shp>, "latin1", None, "ESRI Shapefile")
if writer == QgsVectorFileWriter.NoError:
    print ("success")

QgsApplication.exitQgis()

Answer 1

I think you have to define the source Codepage (Encoding).

# -*- coding: utf-8 -*-
import osgeo.ogr as ogr
import osgeo.osr as osr

OrgShpDat="C:/.../utf8.shp"
OrgCodePage="utf8"
TargetShpDat="C:/.../latin.shp"
TargetCodePage="latin1"

oLayer=QgsVectorLayer(OrgShpDat,None, 'ogr')
oLayer.setProviderEncoding(OrgCodePage)
oLayer.dataProvider().setEncoding(OrgCodePage)

zLayer=QgsVectorFileWriter.writeAsVectorFormat(oLayer,TargetShpDat,TargetCodePage, oLayer.crs(), "ESRI Shapefile")