Is there a way to determine the charset used for a given shapefile?
asked Jul 15, 2011 at 0:59
Add a comment
|
4 Answers
There are two ways for programs to determinate the character set for a shapefile.
- The first is the codepage file *.cpg.
- The second is the codepage info, which can be inside the dbase file. The specification could be found at http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT
answered Jul 19, 2011 at 17:55
-
+1 That link to a dBase file format page is great. However, AFAIK, codepages were never included in the dBase III format. The reference there is to a FoxPro extension of the format, which suggests not all .dbf files are going to have codepage info in them (or, if they do, it might be a result of garbage bytes appearing in a free area of the header). But if you can dig this information out, it's still a good start for a trial-and-error search. BTW, welcome to our site!– whuberCommented Jul 20, 2011 at 13:33
-
Some python dbf libraries can read the codepage if information is there. Commented Aug 6, 2012 at 19:19
Trial and error. Try to open the .dbf file with Ms Excel or with OpenOffice using different setting until you get everything right.
Look at this post for more clues: https://stackoverflow.com/questions/319095/how-do-i-determine-the-character-set-of-a-string
-
If nothing is known about the encoding, it's worthwhile to try latin1 or UTF-8 first.– krlmlrCommented Mar 29, 2015 at 16:00
-
I have opened the .dbf in Ms Excel and the characters show up normally. How can I see which encoding Excel is using/detecting? So that I can set it in QGIS... Commented May 24, 2018 at 18:36
The file utility is able to guess the encoding of a text file. Use ogr2ogr for a conversion that preserves the original encoding if there is no .cpg file:
ogr2ogr -f CSV file.csv file.dbf
file file.csv
Example output:
file.csv: ISO-8859 text
I have tested it with two of the most frequent encodings, UTF-8 and latin1. Works out of the box in Ubuntu, not sure about OS X. I'm not aware of a file utility on Windows.
NOTE: As soon as there is a corresponding .cpg file that indicates the encoding, ogr2ogr will honor it and the output will be in UTF-8. But if the CSV output looks right, you know that the information in the .cpg file is accurate.
Another table for converting 29th byte of *.dbf to code page: http://webhelp.esri.com/arcpad/8.0/referenceguide/index.htm#locales/task_code.htm
