Is there a way to determine the charset used for a given shapefile?
There are two ways for programs to determinate the character set for a shapefile.
- The first is the codepage file *.cpg.
- The second is the codepage info, which can be inside the dbase file. The specification could be found at http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT
Trial and error. Try to open the .dbf file with Ms Excel or with OpenOffice using different setting until you get everything right.
Look at this post for more clues: https://stackoverflow.com/questions/319095/how-do-i-determine-the-character-set-of-a-string
file utility is able to guess the encoding of a text file. Use
ogr2ogr for a conversion that preserves the original encoding if there is no
ogr2ogr -f CSV file.csv file.dbf
file.csv: ISO-8859 text
I have tested it with two of the most frequent encodings, UTF-8 and latin1. Works out of the box in Ubuntu, not sure about OS X. I'm not aware of a
file utility on Windows.
NOTE: As soon as there is a corresponding
.cpg file that indicates the encoding,
ogr2ogr will honor it and the output will be in UTF-8. But if the CSV output looks right, you know that the information in the
.cpg file is accurate.
Another table for converting 29th byte of *.dbf to code page: http://webhelp.esri.com/arcpad/8.0/referenceguide/index.htm#locales/task_code.htm