0

Question

How to change the encoding setting for a spatialpolygon loaded in R without reloading it? Similar to the argument encoding=UTF-8 when loading the file with readOGR or similar to the drop-down setting in QGIS to change the encoding that QGIS uses to read the file. (I don't need to change the encoding of the file itself as it is saved, just able to read the characters correctly)

Also, what is the most efficient (fast) package/command to load large geodatabase .gdb files?

Details

I have a very large geodatabase file that I want to read in R. Available here: https://www.donneesquebec.ca/recherche/fr/dataset/adresses-quebec, although I don't think the file is a problem. The documentation for the file states that it was in Latin-1 ("Le jeu de caractères sélectionné (le « code page ») est le Windows Latin-1 (windows 1252)"). However, the characters did not load properly with latin1(Régionale vs. Régionale) and it turned out that UTF-8 was the correct encoding. I had to reload the file each time to try a different encoding and given that the file is large, it was problematic because it look a long time.

#load layer from geodatabase with latin-1
readOGR(dsn=fgbd, layer="Reseau_routier", encoding="latin1", use_iconv=FALSE)
#load layer from geodatabase with 
readOGR(dsn=fgbd, layer="Reseau_routier", encoding="UTF-8", use_iconv=FALSE)
1

A few things to try:

If you haven't upgraded to R>3.5 yet, you might want to - the documentation ?Encoding notes enhanced behaviour in Windows environments. rgdal::readOGR() may then pick the correct encoding without you having to tell it.

Then again, I'm on Win 10, R 3.5.1, and that GDB layer imported with the correct encoding in ~60 seconds using sf::read_sf(), whereas rgdal::readOGR() just sat there for 10 minutes and then ran out of RAM.

library(sf)

bgdb <- 'C:/Data/geospatial/AQreseauPlus_FGDB/AQreseauPlus.gdb'
rrf <- sf::read_sf(bgdb, 'Reseau_routier')

read_sf is handy for large datasets too, because you can limit import volume with SQLite and check that everything looks ok before reading in all the data:

# first 100 records
rrf_subset <- read_sf(bgdb, 'Reseau_routier', 
                      query = "select * from Reseau_routier limit 100")

To speed up import even more, you could saveRDS() once you've read the dataset in, and work with that going forward.

saveRDS(rrf, 'C:/data/rrf.Rds')
rrf <- readRDS('C:/data/rrf.Rds') # about 15s

If you're still having encoding problems after that, you can find your character columns and convert them like this:

library(dplyr)
dat_u8 <- dat_subset %>%
  dplyr::mutate_if(is.character, iconv, to = 'UTF-8' )

and if you still need to work with an sp object,

library(sp)
dat_sp <- as(dat_subset, 'Spatial')
dat_u8 <- dat_sp
dat_u8@data <-  dat_u8@data %>%
  dplyr::mutate_if(is.character, iconv, to = 'UTF-8')
|improve this answer|||||
  • I have also made a few tests and read_sfautomatically detects the encoding for shapefiles whereas as shapefile from the raster package does not. Very useful! – user3386170 Dec 11 '18 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.