Python - read shapefile .dbf for encoding

Question

I've written a python script that formats a .shp such that its fields structure matches my office's database view in order to ingest the data of said shapefile. This original shapefile was encoded with the regional Hangul (Wansung) encoding, which required the codecs.decode('949') method to read in the Korean characters. This is consistent with the ESRI table found here.

What I'd like to do is enhance the script so that it can read the .dbf header, find the encoding, which according to this and other sites resides at the byte offset of 29, so that our staff does not have to know the ESRI language code for every regionally-encoded shapefile we get. It's a one-byte flag, and I figured '949' would be an integer, so python's struct.unpack method should do the trick, specifically this:

myBinaryFile = angusMcFile.read(30)
dbfHead = struct.unpack('<29xB', myBinaryfile)

The returned values are 'W' when I unpack guessing that this 'flag' is a 1-byte string, and '87' when I unpack guessing it's a 1-byte integer. This is not the '949' or '0x79' that I'm looking for. What am I doing wrong? I'll be happy to provide more code if necessary.

Answer 1

87 (0x57) is the ANSI code page ID according to this reference.

dat=open(dbf,'rb').read(30)[29:]
id=struct.unpack('B',dat)[0]
print id,hex(id),chr(id)

This prints 87 0x57 W on some random dbf I tested. For '949' you should be looking for 78 0x4E.

Edit: Below is a look up table (dict) for the code pages copied from the above reference:

lut={
        1    :['437','US MS-DOS'],
        2    :['850','International MS-DOS'],
        3    :['1252','Windows ANSI Latin I'],
        4    :['10000','Standard Macintosh'],
        8    :['865','Danish OEM'],
        9    :['437','Dutch OEM'],
        10   :['850','Dutch OEM'],
        11   :['437','Finnish OEM'],
        13   :['437','French OEM'],
        14   :['850','French OEM'],
        15   :['437','German OEM'],
        16   :['850','German OEM'],
        17   :['437','Italian OEM'],
        18   :['850','Italian OEM'],
        19   :['932','Japanese Shift-JIS'],
        20   :['850','Spanish OEM'],
        21   :['437','Swedish OEM'],
        22   :['850','Swedish OEM'],
        23   :['865','Norwegian OEM'],
        24   :['437','Spanish OEM'],
        25   :['437','English OEM (Great Britain)'],
        26   :['850','English OEM (Great Britain)'],
        27   :['437','English OEM (US)'],
        28   :['863','French OEM (Canada)'],
        29   :['850','French OEM'],
        31   :['852','Czech OEM'],
        34   :['852','Hungarian OEM'],
        35   :['852','Polish OEM'],
        36   :['860','Portuguese OEM'],
        37   :['850','Portuguese OEM'],
        38   :['866','Russian OEM'],
        55   :['850','English OEM (US)'],
        64   :['852','Romanian OEM'],
        77   :['936','Chinese GBK (PRC)'],
        78   :['949','Korean (ANSI/OEM)'],
        79   :['950','Chinese Big5 (Taiwan)'],
        80   :['874','Thai (ANSI/OEM)'],
        87   :['Current ANSI CP','ANSI'],
        88   :['1252','Western European ANSI'],
        89   :['1252','Spanish ANSI'],
        100  :['852','Eastern European MS-DOS'],
        101  :['866','Russian MS-DOS'],
        102  :['865','Nordic MS-DOS'],
        103  :['861','Icelandic MS-DOS'],
        104  :['895','Kamenicky (Czech) MS-DOS'],
        105  :['620','Mazovia (Polish) MS-DOS'],
        106  :['737','Greek MS-DOS (437G)'],
        107  :['857','Turkish MS-DOS'],
        108  :['863','French-Canadian MS-DOS'],
        120  :['950','Taiwan Big 5'],
        121  :['949','Hangul (Wansung)'],
        122  :['936','PRC GBK'],
        123  :['932','Japanese Shift-JIS'],
        124  :['874','Thai Windows/MS–DOS'],
        134  :['737','Greek OEM'],
        135  :['852','Slovenian OEM'],
        136  :['857','Turkish OEM'],
        150  :['10007','Russian Macintosh'],
        151  :['10029','Eastern European Macintosh'],
        152  :['10006','Greek Macintosh'],
        200  :['1250','Eastern European Windows'],
        201  :['1251','Russian Windows'],
        202  :['1254','Turkish Windows'],
        203  :['1253','Greek Windows'],
        204  :['1257','Baltic Windows']
}