Reading KML file into R?

Question

I'm working with huge .kml files (up to 10 Gb) and need an efficient way to read them into R. Until now I've been converting them to shapefiles via QGIS and then back into R with readShapePoly and readOGR (the latter, by the way, is ~1000 faster than the former). I'd ideally like to cut-out the QGIS intermediary stage as it is cumbersome and slow.

How to read .kml files in directly?

I see this can be also be done with readOGR. Unfortunately, I cannot see how to implement the worked example (after lengthy preparation of .kml file: xx <- readOGR(paste(td, "cities.kml", sep="/"), "cities")). It seems that "cities" here is the name of the spatial objects.

Roger Bivand admits that "How one discovers this name is not obvious, since the KML driver in OGR needs it to access the file. One possibility is:

system(paste("ogrinfo", paste(td, "cities.kml", sep="/")), intern=TRUE)

"

But this does not work for me either. Here's a test .kml file to try it on. With it in my working directory, readOGR("x.kml", "id") generates this error message:

Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv = use_iconv) : 
  Cannot open layer . 

And system(paste("ogrinfo", "x.kml"), intern=TRUE) generates:

[1] "Had to open data source read-only."   "INFO: Open of `x.kml'"               
[3] "      using driver `KML' successful." "1: x (3D Polygon)"  

, which I simply don't understand.

Would getKMLcoordinates {maptools} be a valid alternative?

I've also tried this:

tkml <- getKMLcoordinates(kmlfile="x.kml", ignoreAltitude=T)
head(tkml[[1]])
tkml <- SpatialPolygons(tkml, 
                        proj4string=CRS("+init=epsg:3857"))

The coordinates are generated correctly, but my attempt to convert them back into a polygon object failed with the following message:

Error in SpatialPolygons(tkml, proj4string = CRS("+init=epsg:3857")) : 
  cannot get a slot ("area") from an object of type "double"

Answer 1

To read a KML with the OGR driver, you give it the file name and the layer name.

Roger's comment is that the layer name is hidden in the KML file, and unless you know how the KML was created you can't infer the layer name from the KML file name.

Looking at your example KML, I can see:

<?xml version="1.0" encoding="utf-8" ?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document><Folder><name>x</name>
<Schema name="x" id="x">

Which is telling me the layer name is x, not id, and so:

> foo = readOGR("/tmp/x.kml", "x")
OGR data source with driver: KML 
Source: "/tmp/x.kml", layer: "x"
with 1 features and 2 fields
Feature type: wkbPolygon with 2 dimensions

works nicely.

Now, you can try and get the name by parsing the KML as XML using an R XML parser, or you can maybe try reading it in R as a text file until you find the name tag.

The other approach is to run the command-line ogrinfo program which spits out the layer names of a KML file:

$ ogrinfo /tmp/x.kml 
Had to open data source read-only.
INFO: Open of `/tmp/x.kml'
      using driver `KML' successful.
1: x (Polygon)

here showing there is a polygon layer called x.

Answer 2

If you want to do the alternative way using maptool, this should work:

tkml <- getKMLcoordinates(kmlfile="yourkml.kml", ignoreAltitude=T)
#make polygon
p1 = Polygon(tkml)
#make Polygon class
p2 = Polygons(list(p1), ID = "drivetime")
#make spatial polygons class
p3= SpatialPolygons(list(p2),proj4string=CRS("+init=epsg:4326"))

The key here is you need to go through a couple steps to make spatial polygon class.

Answer 3

Don't know if this is still a problem for anybody else, but I was running in circles for a while with this. What finally worked for me is below. It uses the XML package to get at the xmlValue of the right node. I had to set the layer parameter of readOGR to the name of the one of the folders within the kml file. When I set the layer parameter to the of the kml file, I would get the same error that RobinLovelace described above.

Shown below are a lot of code lines that only show how to see the various node levels of the kml document. I think this will be slightly different depending on the source of the kml. But you should be able to use the same logic to determine the correct parameter value.

Also, I created a list of kml files so it could be easily made into a function that could be put in an lapply - do.call pair. This could then pull in a data from a long list of kml files. Or, a lot of subfolders within a single kml file as it seems readOGR cannot deal with multiple subfolders in a kml file.

library(rgdal); library(XML)

# SET WORKING DIRECTORY FIRST!!
dir <- getwd()

kmlfilelist <- list.files(dir, pattern =".kml$", full.names=TRUE, recursive=FALSE)

doc0 <- xmlTreeParse(kmlfilelist[2], useInternal = TRUE)
rootNode0 <- xmlRoot(doc0)
rootName0 <- xmlName(rootNode0)
element1Name0 <- names(rootNode0)

nodeNames <- names(rootNode0[1][[1]])

# entire rootNode - kml Document level
rootNode0[[1]]

# 1st element of rootNode - kml file name
rootNode0[[1]][[1]] 

# 2nd element of rootNode - kml Style Map 
rootNode0[[1]][[2]] 

# 3rd element of rootNode - Style
rootNode0[[1]][[3]]

# 4th element of rootNode - Style
rootNode0[[1]][[4]] 

# 5th element of rootNode - kml Folder with data in it.
rootNode0[[1]][[5]] 

# 5th element 1st subelement of rootNode - kml Folder name with data in it. 
#  What to set readOGR() layer parameter to.
rootNode0[[1]][[5]][[1]] 

kmlfoldername <- xmlValue(rootNode0[[1]][[5]][[1]]) # Folder name to set = layer.

readOGR(dsn=kmlfilelist[2], layer =  kmlfoldername)

Answer 4

Don't know if I should have modified my previous answer. Perhaps, but that covers some things not in this answer, so I decided to leave it.

Anyway, the code below works well for me. It looks for all of xmlNodes in the kml file that are called "Folder" and then sets the layer parameter of readOGR to that xmlValue. Tested on working directory with around 6 separate kml files. Output is a list of imported SpatialDataFrames objects. Each SpatialDataFrame can easily be subset from the list.

Still does not address kml files with multiple Folder nodes. But that feature could easily be added with another nested apply function.

library(rgdal); library(XML)

# SET WORKING DIRECTORY FIRST!!
dir <- getwd()

kmlfilelist <- list.files(dir, pattern =".kml$", full.names=TRUE, recursive=FALSE)

ImportKml <- function (kmlfile) {
  doc0 <- xmlTreeParse(kmlfile, useInternal = TRUE)
  rootNode0 <- xmlRoot(doc0)
  rootName0 <- xmlName(rootNode0)
  element1Name0 <- names(rootNode0)

  kmlNodeNames <- unname(names(rootNode0[1][[1]]))
  kmlFolderNodeNum <- which(kmlNodeNames == "Folder")
  kmlFolderNodeName <- xmlValue(rootNode0[[1]][[kmlFolderNodeNum]][[1]])

  kmlIn <- readOGR(dsn=kmlfile, layer = kmlFolderNodeName)
}
ImportedKmls <- lapply(kmlfilelist, ImportKml)