5

Long story short, I'm trying to import this ESRI shapefile of Japan into CartoDB. (Sorry, no direct link: to download, click on the orange ファイルのダウンロード button, check 同意する to agree to the T&C, then click on the green 全国市区町村界データのダウンロード button.)

Problem is, the DBF in the file is encoded as Shift_JIS, and CartoDB only likes UTF-8. I've tried the following unsuccessfully:

1) ogr2ogr

ogr2ogr --config SHAPE_ENCODING Shift_JIS japan_ver72_utf8.shp 

No-op: SJIS in, SJIS out.

ogr2ogr --config SHAPE_ENCODING UTF-8 japan_ver72_utf8 japan_ver72.shp

Makes ogr2ogr think the input is UTF-8, meaning I get garbage out.

2) QGIS

Load the shapefile into QGIS as ShiftJIS. But while the shapes load fine, QGIS dumps a whole bunch of this on load:

ERROR 1: fread(48623) failed on DBF file.

And inspecting the attribute table just shows a bunch of nulls, so there's no point trying to save as UTF-8.

3) OpenOffice Calc

Load the DBF into OpenOffice, re-export as SJIS. But OO throws an error when parsing the DBF and refuses to import the file at all.

4) iconv

Run iconv directly on the DBF:

iconv -f Shift_JIS -t UTF-8 japan_ver72_sjis.dbf >japan_ver72.dbf

This "works", in the sense that the Japanese within is correctly recoded as UTF-8, but it destroys the DBF in the process.

Ideas?

  • Have you already tried to convert directly from shp into CartoDB? Syntax from gdal.org/drv_cartodb.html ogr2ogr --config CARTODB_API_KEY abcdefghijklmnopqrstuvw --config SHAPE_ENCODING Shift_JIS -f CartoDB "CartoDB:myaccount" myshapefile.shp – user30184 Apr 21 '15 at 10:56
  • @user30184 I'm on GDAL 1.10.1, which doesn't appear to include the CartoDB driver. – lambshaanxy Apr 23 '15 at 4:32
2
+50

If you only need to do the job once and there is no need to go to scripting then one simple way is to convert the data with OpenJUMP.

Activate the charactes set selection from menu Customize - Options

enter image description here

Open your dataset as Shift-JIS

enter image description here

Save data back with Save as... and select UTF-8 charset

enter image description here

  • Whoever came up with the UI for this program should be taken out and shot, but on the upside, it works! One minor nit: this process drops the .prj file along the way, so you need to manually copy it back in. The bountry is still yours unless someone comes up with a more elegant way of doing this. – lambshaanxy Apr 23 '15 at 4:52
  • List of developers is in this readme sourceforge.net/p/jump-pilot/code/HEAD/tree/core/trunk/etc/…. OpenJUMP is unaware of projections and therefore it does not read or write .prj files. – user30184 Apr 23 '15 at 5:20
3

Recently, i also encounter problems with Chinese character read in dbf file !

Here is the convert tool for shapefile to geojson via web browser without server-side code and supporting non-english encoding, just need to upload the zip file and set the encoding (Shift_JIS) for the correctly display Japanese text.

http://gipong.github.io/shp2geojson.js/

enter image description here

It will create a geojson file , so you can use with leaflet.js, openlayer or cartodb.js. https://github.com/gipong/shp2geojson.js

  • +1 for an interesting option, but I was asking about a way to fix a shapefile, not convert it to GeoJSON. – lambshaanxy Apr 24 '15 at 4:26
  • Sorry about that, i focus on how to covert the encoding to utf-8 and make it can import to CartoDB... Do you mean how to open the dbf correctly in some method? – Gipong Apr 24 '15 at 7:36
1

ogr2ogr man page says that this should work

ogr2ogr --config SHAPE_ENCODING Shift_JIS japan_ver72_utf8.shp  -lco ENCODING=UTF-8

Have you already tried it ? (Probably needs linux version of ogr2ogr)

  • 1
    I do not believe that -lco ENCODING=UTF-8 will convert the data. Rather is just writes the given encoding into the dbf headers. However, I may be wrong. – user30184 Apr 21 '15 at 11:00
  • Sorry, doesn't work: ERROR 1: fread(48623) failed on DBF file. (on Ubuntu) – lambshaanxy Apr 23 '15 at 4:23
1

Using the version 1.2 of H2GIS (java spatial database) here: http://search.maven.org/remotecontent?filepath=org/orbisgis/h2-dist/1.2.0/h2-dist-1.2.0-bin.zip

You can load the file into the database using the right encoding. H2GIS is able to find the right encoding in the dbf header, however under windows the encoding name is not supported.

Unzip the file. Run with last java the file h2-dist-1.2.0.jar

Select embedded database and connect.

You have to spatially enable the database:

CREATE ALIAS IF NOT EXISTS SPATIAL_INIT FOR
    "org.h2gis.h2spatialext.CreateSpatialExtension.initSpatialExtension";
CALL SPATIAL_INIT();

Then import the shapefile:

-- If under windows
CALL SHPREAD('file:///E:/downloads/japan_ver72/japan_ver72.shp','JAPAN_VER72','Shift_JIS');
-- if under linux/mac
CALL SHPREAD('/home/user/downloads/japan_ver72/japan_ver72.shp')

Finally export the table into a shapefile under the UTF-8 encoding.

CALL SHPWRITE('file:///E:/downloads/japan_ver72/converted.shp', 'JAPAN_VER72', 'utf-8');

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.