Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I've found several example of script and blog posts on working with unicode characters, but I haven't been able to make anything works so far... It's a little frustrating

I have a shapefile encoded in utf-8 (exported from QGIS) and there's many É, È, À, Ô, etc. in the values of some fields.

For a geocoder, I need to normalize my fields values.

This is what I have so far:

# -*- coding: utf-8 -*-
###############################################################################
import sys
from sys import argv
import osgeo.ogr
from amtpy import EndOfScript
###############################################################################

script, src_shp, fld_shp = argv

#- Opening the shapefile
shapefile = osgeo.ogr.Open(src_shp)
layer = shapefile.GetLayer(0)
spatialRef = layer.GetSpatialRef()


#- Going through each feature, one by one
for i in range(layer.GetFeatureCount()):
    print "Normalisation de la ligne %i" %(i+1)
    feat = layer.GetFeature(i)

    texte_norm = feat.GetField(fld_shp)
    texte_norm = texte_norm.encode('utf-8')

    texte_norm = texte_norm.upper()
    texte_norm = texte_norm.replace(u'\00c2', 'A') #À
    texte_norm = texte_norm.replace(u'\u00C9', 'E') #É
    texte_norm = texte_norm.replace(u'\u00C8', 'E') #È
    # I've remove 16-17 characters to replace, for the example...
    texte_norm = texte_norm.decode('utf-8')

    print texte_norm
    #feat.SetField(fld_shp, texte_norm)

#- print EndOfScript message
print EndOfScript()

I've put a print instead of the SetField just so I could see if the replace works.

Then the script encounters the first unicode character, I have this error message: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128).

Well, I know that, since the character is a "À". I can't seem to find the right way.

Any tips?

Thanks!

share|improve this question
    
What are you trying to do exactly? Transliterate the string attributes of features from UTF-8 to ASCII? Are you interested in an example of how I would do it with Fiona and unidecode? –  sgillies Apr 22 '13 at 20:40
1  
I think that if you remove the texte_norm.encode('utf-8') step and just skip straight to the replace function it should work: >>> s = 'Ècouter' >>> s '\xc3\x88couter' >>> print s.replace('\xc3\x88','E') Ecouter –  Jason Apr 22 '13 at 20:58
    
I remove any encoding/decoding and used «texte_norm = texte_norm.replace('\xc3\x80', 'A') #À» etc. It didn't crash, but there's must be something wrong with my « feat.SetField(fld_shp, texte_norm)» (uncommented as from my example of course), because my shapefile's dbf didn't change... Thanks –  fgcartographix Apr 23 '13 at 16:07
1  
You have to open your shapefile in update mode to modify it. You've opened it in read-only mode. –  sgillies Apr 23 '13 at 16:21
    
That would be with «shapefile = osgeo.ogr.Open(src_shp, 1)»? Still nothing. Thanks for the help, I'll keep looking. –  fgcartographix Apr 23 '13 at 17:13
add comment

1 Answer

up vote 5 down vote accepted

GetField returns UTF-8 encoded strings and you'll want to decode it before you process it in any way. Then you encode the result to pass it to SetField. You've got it backwards.

Fiona (shameless plug) deals in Python unicode strings and so is simpler to use.

Unidecode (https://pypi.python.org/pypi/Unidecode) is handy for stuff like this because it will make sensible transliterations and romanizations for many languages. It looks like it would make the ones you want.

>>> from unidecode import unidecode
>>> unidecode(u'\u00c2')
'A'
>>> unidecode(u'\u00C9')
'E'
>>> unidecode(u'\u00C8')
'E'

The example below uses Natural Earth data and converts "Côte d'Ivoire" to "Cote d'Ivoire", etc, without presuming anything about the characters in the source data.

import fiona
from unidecode import unidecode

with fiona.open(
        '/Users/seang/data/ne_50m_admin_0_countries/'
        'ne_50m_admin_0_countries.shp', 'r') as source:

    # Create an output shapefile with the same schema,
    # coordinate systems. ISO-8859-1 encoding.
    with fiona.open(
            '/tmp/transliterated.shp', 'w',
            **source.meta) as sink:

        # Identify all the str type properties.
        str_prop_keys = [
            k for k, v in sink.schema['properties'].items()
                if v.startswith('str')]

        for rec in source:

            # Transliterate and update each of the str properties.
            for key in str_prop_keys:
                val = rec['properties'][key]
                if val:
                    rec['properties'][key] = unidecode(val)

            # Write out the transformed record.
            sink.write(rec)
share|improve this answer
    
Thanks Sean! I've been looking to take a look at Fiona for a long time. Shapely is so useful, it's a great complement! Now, how to make those string uppercase? ;) –  fgcartographix Apr 23 '13 at 17:15
    
I have other string modification to make, remove all "-" in names, change all "Saint-" and "Sainte-" into "S.". Is it possible? –  fgcartographix Apr 23 '13 at 17:26
    
For what I understand of your code, I think I could use the model of «rec['properties'][key] = unidecode(val)», I could do «rec['properties'][key] = val.replace('-', ' ')» after the unidecode command... Trying right now! (just so you know I don't want the answer without doing my part! ;) ) –  fgcartographix Apr 23 '13 at 18:39
    
Nope. I've made several tests and I still have streets like "RUE LéOTABLE-DUBUC". The é isn't removed... –  fgcartographix Apr 23 '13 at 19:03
    
The é are still there only when I try to do something else to (val). Your code works just fine as it is... –  fgcartographix Apr 24 '13 at 14:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.