Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I've written a python script that formats a .shp such that its fields structure matches my office's database view in order to ingest the data of said shapefile. This original shapefile was encoded with the regional Hangul (Wansung) encoding, which required the codecs.decode('949') method to read in the Korean characters. This is consistent with the ESRI table found here.

What I'd like to do is enhance the script so that it can read the .dbf header, find the encoding, which according to this and other sites resides at the byte offset of 29, so that our staff does not have to know the ESRI language code for every regionally-encoded shapefile we get. It's a one-byte flag, and I figured '949' would be an integer, so python's struct.unpack method should do the trick, specifically this:

myBinaryFile = angusMcFile.read(30)
dbfHead = struct.unpack('<29xB', myBinaryfile)

The returned values are 'W' when I unpack guessing that this 'flag' is a 1-byte string, and '87' when I unpack guessing it's a 1-byte integer. This is not the '949' or '0x79' that I'm looking for. What am I doing wrong? I'll be happy to provide more code if necessary.

share|improve this question
    
One single byte of information could only ever hold numbers from 0-255 (or -128-127 if signed). The field 32-63- 32 bytes - Language driver name looks a lot more promising. –  Jason Scheirer Feb 22 '13 at 3:27
add comment

1 Answer 1

up vote 3 down vote accepted

87 (0x57) is the ANSI code page ID according to this reference.

dat=open(dbf,'rb').read(30)[29:]
id=struct.unpack('B',dat)[0]
print id,hex(id),chr(id)

This prints 87 0x57 W on some random dbf I tested. For '949' you should be looking for 78 0x4E.

Edit: Below is a look up table (dict) for the code pages copied from the above reference:

lut={
        1    :['437','US MS-DOS'],
        2    :['850','International MS-DOS'],
        3    :['1252','Windows ANSI Latin I'],
        4    :['10000','Standard Macintosh'],
        8    :['865','Danish OEM'],
        9    :['437','Dutch OEM'],
        10   :['850','Dutch OEM'],
        11   :['437','Finnish OEM'],
        13   :['437','French OEM'],
        14   :['850','French OEM'],
        15   :['437','German OEM'],
        16   :['850','German OEM'],
        17   :['437','Italian OEM'],
        18   :['850','Italian OEM'],
        19   :['932','Japanese Shift-JIS'],
        20   :['850','Spanish OEM'],
        21   :['437','Swedish OEM'],
        22   :['850','Swedish OEM'],
        23   :['865','Norwegian OEM'],
        24   :['437','Spanish OEM'],
        25   :['437','English OEM (Great Britain)'],
        26   :['850','English OEM (Great Britain)'],
        27   :['437','English OEM (US)'],
        28   :['863','French OEM (Canada)'],
        29   :['850','French OEM'],
        31   :['852','Czech OEM'],
        34   :['852','Hungarian OEM'],
        35   :['852','Polish OEM'],
        36   :['860','Portuguese OEM'],
        37   :['850','Portuguese OEM'],
        38   :['866','Russian OEM'],
        55   :['850','English OEM (US)'],
        64   :['852','Romanian OEM'],
        77   :['936','Chinese GBK (PRC)'],
        78   :['949','Korean (ANSI/OEM)'],
        79   :['950','Chinese Big5 (Taiwan)'],
        80   :['874','Thai (ANSI/OEM)'],
        87   :['Current ANSI CP','ANSI'],
        88   :['1252','Western European ANSI'],
        89   :['1252','Spanish ANSI'],
        100  :['852','Eastern European MS-DOS'],
        101  :['866','Russian MS-DOS'],
        102  :['865','Nordic MS-DOS'],
        103  :['861','Icelandic MS-DOS'],
        104  :['895','Kamenicky (Czech) MS-DOS'],
        105  :['620','Mazovia (Polish) MS-DOS'],
        106  :['737','Greek MS-DOS (437G)'],
        107  :['857','Turkish MS-DOS'],
        108  :['863','French-Canadian MS-DOS'],
        120  :['950','Taiwan Big 5'],
        121  :['949','Hangul (Wansung)'],
        122  :['936','PRC GBK'],
        123  :['932','Japanese Shift-JIS'],
        124  :['874','Thai Windows/MS–DOS'],
        134  :['737','Greek OEM'],
        135  :['852','Slovenian OEM'],
        136  :['857','Turkish OEM'],
        150  :['10007','Russian Macintosh'],
        151  :['10029','Eastern European Macintosh'],
        152  :['10006','Greek Macintosh'],
        200  :['1250','Eastern European Windows'],
        201  :['1251','Russian Windows'],
        202  :['1254','Turkish Windows'],
        203  :['1253','Greek Windows'],
        204  :['1257','Baltic Windows']
}
share|improve this answer
    
Thanks for finding and sharing this, very good reference. –  Jason Scheirer Feb 22 '13 at 22:04
    
Thank you for answering this - I tested it out on the .dbf at the office and '78' resulted. Quite a relief! –  Foszter Feb 28 '13 at 15:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.