Converting shapefile from Shift_JIS to UTF-8 when the usual methods fail

Question

Long story short, I'm trying to import this ESRI shapefile of Japan into CartoDB. (Sorry, no direct link: to download, click on the orange ファイルのダウンロード button, check 同意する to agree to the T&C, then click on the green 全国市区町村界データのダウンロード button.)

Problem is, the DBF in the file is encoded as Shift_JIS, and CartoDB only likes UTF-8. I've tried the following unsuccessfully:

1) ogr2ogr

ogr2ogr --config SHAPE_ENCODING Shift_JIS japan_ver72_utf8.shp 

No-op: SJIS in, SJIS out.

ogr2ogr --config SHAPE_ENCODING UTF-8 japan_ver72_utf8 japan_ver72.shp

Makes ogr2ogr think the input is UTF-8, meaning I get garbage out.

2) QGIS

Load the shapefile into QGIS as ShiftJIS. But while the shapes load fine, QGIS dumps a whole bunch of this on load:

ERROR 1: fread(48623) failed on DBF file.

And inspecting the attribute table just shows a bunch of nulls, so there's no point trying to save as UTF-8.

3) OpenOffice Calc

Load the DBF into OpenOffice, re-export as SJIS. But OO throws an error when parsing the DBF and refuses to import the file at all.

4) iconv

Run iconv directly on the DBF:

iconv -f Shift_JIS -t UTF-8 japan_ver72_sjis.dbf >japan_ver72.dbf

This "works", in the sense that the Japanese within is correctly recoded as UTF-8, but it destroys the DBF in the process.

Ideas?

Answer 1

Recently, i also encounter problems with Chinese character read in dbf file !

Here is the convert tool for shapefile to geojson via web browser without server-side code and supporting non-english encoding, just need to upload the zip file and set the encoding (Shift_JIS) for the correctly display Japanese text.

http://gipong.github.io/shp2geojson.js/

It will create a geojson file , so you can use with leaflet.js, openlayer or cartodb.js. https://github.com/gipong/shp2geojson.js

Answer 2

If you only need to do the job once and there is no need to go to scripting then one simple way is to convert the data with OpenJUMP.

Activate the charactes set selection from menu Customize - Options

Open your dataset as Shift-JIS

Save data back with Save as... and select UTF-8 charset

Answer 3

ogr2ogr man page says that this should work

ogr2ogr --config SHAPE_ENCODING Shift_JIS japan_ver72_utf8.shp  -lco ENCODING=UTF-8

Have you already tried it ? (Probably needs linux version of ogr2ogr)

Answer 4

Using the version 1.2 of H2GIS (java spatial database) here: http://search.maven.org/remotecontent?filepath=org/orbisgis/h2-dist/1.2.0/h2-dist-1.2.0-bin.zip

You can load the file into the database using the right encoding. H2GIS is able to find the right encoding in the dbf header, however under windows the encoding name is not supported.

Unzip the file. Run with last java the file h2-dist-1.2.0.jar

Select embedded database and connect.

You have to spatially enable the database:

CREATE ALIAS IF NOT EXISTS SPATIAL_INIT FOR
    "org.h2gis.h2spatialext.CreateSpatialExtension.initSpatialExtension";
CALL SPATIAL_INIT();

Then import the shapefile:

-- If under windows
CALL SHPREAD('file:///E:/downloads/japan_ver72/japan_ver72.shp','JAPAN_VER72','Shift_JIS');
-- if under linux/mac
CALL SHPREAD('/home/user/downloads/japan_ver72/japan_ver72.shp')

Finally export the table into a shapefile under the UTF-8 encoding.

CALL SHPWRITE('file:///E:/downloads/japan_ver72/converted.shp', 'JAPAN_VER72', 'utf-8');