Character encoding support in geodatabases and shapefiles

Question

I've got several geodatabases that include feature classes with Greek letters in a lot of the attributes. When I try exporting a feature class as a shapefile from ArcCatalog, the attributes are butchered in the shapefile data, some kind of character encoding problem (they look like this in the shape: ??etr??e?). The same thing happens when I use ogr2ogr in FWtools to convert layers from the MDB to KML, shp, etc.

Does anyone have experience trying to deal with encoding formats across GIS data formats?

The real goal here is to get some data from these Esri geodatabases into a Postgres/PostGIS database, but broken encoding isn't going to work. I was going to export from the geoDBs to shapefiles, then load them in with shp2pgsql. Is that the easiest path to get there?

Answer 1

I think you're part way there. You can use iconv to convert from one encoding to another, and you can use this as part of the shp2pgsql process. For example:

shp2pgsql *postgrestablename* | iconv -f *sourceencoding* -t *targetencoding* | psql -d *yourdatabase*

If you're working in a Linux environment then iconv should be installed already. For Windows I found LibIconv for Windows. But I have no experience of using iconv under Windows, so I can't vouch for it.

Hope this helps!

Jo

Answer 2

You'll need to first figure out what encoding the input data is in, so you can tell your tools how to convert the data into an appropriate encoding. If you have Access, I'd try exporting the table to text directly from the MDB and set the output encoding to UTF8. If you open up the exported shapefile in ArcGIS, is the encoding set correctly? DBF supports code pages, and its possible that OGR isn't picking up the correct one for the conversion.

There are also ways of coercing MDBtools (used as part of the OGR driver) to explicitly set the input stream, but I'd try the other approaches first.

Answer 3

I will rather go to ArcGIS way. Just set the encoding to UTF-8 in ArcGIS by following the instruction from here. After this just export the feature classes to ShapeFile. Now you will get an extra CPG(code page file) file with each layer. This is only a text file with string "UTF-8" in it and all your data is encoded into UTF-8 automatically.

If you are interested to use other encoding just see the instructions.

Important thing is after finishing this assignment you should change this setting to default value becuase if you keep this value for example "UTF-8" then in future ArcGIS will export all ShapeFiles using "UTF-8" encoding.

Hope that will help you.