Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I frankly am a bit confused when I am working outside the United States so please excuse my ignorance.

I downloaded some shapefiles for Moldova (MoldovaData) and had a question regarding the encoding of the attributes.

I have other data I want to link to it but many of the city names are encoded differently. For instance I believe the attribute BălÈ›i corresponds to Balti (Bălți) how do I reconcile the attribute names ?

share|improve this question

5 Answers 5

up vote 2 down vote accepted

Shapefiles get their codepage either from the .dbf or from the .cpg file.

The .dbf file has a byte that represents DBF Language Driver ID. There's some discussion about these in an archived ArcGIS Desktop forum on forums.esri.com. There's a Microsoft Knowledge Base article Understanding Code Pages in Visual FoxPro which lists 19 DBF Language Driver IDs and their corresponding codepages.

The ArcGIS Resource Center page for Shapefile file extensions states that the .cpg is an optional file that can be used to specify the codepage for identifying the characterset to be used.

In ArcGIS, if a .cpg file is present it will take precedence over the DBF Language Driver ID in the .dbf file. This is generally preferred because the DBF Language Driver ID covers languages supported during the dBASE IV era whereas the .cpg file supports any codepage.

The Moldova shapefiles are using a UTF-8 encoding. You can only specify UTF-8 encoding using the .cpg file. Therefore you will need to create a .cpg text file for each shapefile and place either 65001 or UTF-8 in its body. For your convenience I've included the following MAKECPG.BAT batch file which you can save and run to create the .cpg files:

REM MAKECPG.BAT
ECHO 65001 > moldova_administrative.cpg
ECHO 65001 > moldova_coastline.cpg
ECHO 65001 > moldova_highway.cpg
ECHO 65001 > moldova_location.cpg
ECHO 65001 > moldova_natural.cpg
ECHO 65001 > moldova_poi.cpg
ECHO 65001 > moldova_water.cpg
share|improve this answer
    
Hi BicycleDude, that seemed to do the trick! Thanks ! The only thing that people should realize is that you also need to change the fonts within ArcGIS to be able to see the the characters correctly. Be aware, you can change the fonts only for the whole table (not one column) but at least you have full control of all the fonts of your labels. I randomly found one that works (Segoe UI). –  Don Feb 3 '12 at 3:06

What application are you using to read the shapefile? Typically the application has a setting for the encoding of strings in the shapefile, because the shapefile itself does not contain encoding information.

share|improve this answer
    
I am using ArcGIS 9.3. –  Don Jan 26 '12 at 19:01

If possible, I would suggest using OSM XML extracts instead if you want to preserve Unicodeness and topology of OSM data. Shapefiles are just PITA.

share|improve this answer

You are correct: the string "Bălți" would have been represented in UTF-8 format inside OpenSteetMap. If mis-interpreted as iso-8859-1 it comes out "BălÈ›i".

You can verify this at my favorite unicode tool: http://www.ltg.ed.ac.uk/~richard/utf-8.cgi

You can get proper XML dumps of OpenSteetMap data here: http://download.geofabrik.de/osm/europe/

And then see here for more on shapefiles and encoding: Which character-encoding has the attribute-table of a Shapefile?

share|improve this answer

You could try to run iconv on your file.

Something like :

iconv -f LATIN1 -t UTF-8 /ptah/to/yourosm.file
share|improve this answer
    
Since .dbf files are not text files, wouldn't this be rather dangerous advice for you to give? This is most likely going to produce a corrupt unusable .dbf file. –  Stephen Quan Jan 29 '12 at 23:09
    
Well, it is worth a try. If data are corrupted after the operation, Don can just download them again from MoldovaData. –  simo Jan 30 '12 at 8:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.