Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working with huge .kml files (up to 10 Gb) and need an efficient way to read them into R. Until now I've been converting them to shapefiles via QGIS and then back into R with readShapePoly and readOGR (the latter, by the way, is ~1000 faster than the former). I'd ideally like to cut-out the QGIS intermediary stage as it is cumbersome and slow.

How to read .kml files in directly?

I see this can be also be done with readOGR. Unfortunately, I cannot see how to implement the worked example (after lengthy preparation of .kml file: xx <- readOGR(paste(td, "cities.kml", sep="/"), "cities")). It seems that "cities" here is the name of the spatial objects.

Roger Bivand admits that "How one discovers this name is not obvious, since the KML driver in OGR needs it to access the file. One possibility is:

system(paste("ogrinfo", paste(td, "cities.kml", sep="/")), intern=TRUE)

"

But this does not work for me either. Here's a test .kml file to try it on. With it in my working directory, readOGR("x.kml", "id") generates this error message:

Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv = use_iconv) : 
  Cannot open layer . 

And system(paste("ogrinfo", "x.kml"), intern=TRUE) generates:

[1] "Had to open data source read-only."   "INFO: Open of `x.kml'"               
[3] "      using driver `KML' successful." "1: x (3D Polygon)"  

, which I simply don't understand.

Would getKMLcoordinates {maptools} be a valid alternative?

I've also tried this:

tkml <- getKMLcoordinates(kmlfile="x.kml", ignoreAltitude=T)
head(tkml[[1]])
tkml <- SpatialPolygons(tkml, 
                        proj4string=CRS("+init=epsg:3857"))

The coordinates are generated correctly, but my attempt to convert them back into a polygon object failed with the following message:

Error in SpatialPolygons(tkml, proj4string = CRS("+init=epsg:3857")) : 
  cannot get a slot ("area") from an object of type "double"
share|improve this question
    
You can get the layers in the kml using rgdal's function ogrListLayers. – Mario Becerra C Nov 24 '15 at 23:21
up vote 22 down vote accepted

To read a KML with the OGR driver, you give it the file name and the layer name.

Roger's comment is that the layer name is hidden in the KML file, and unless you know how the KML was created you can't infer the layer name from the KML file name.

Looking at your example KML, I can see:

<?xml version="1.0" encoding="utf-8" ?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document><Folder><name>x</name>
<Schema name="x" id="x">

Which is telling me the layer name is x, not id, and so:

> foo = readOGR("/tmp/x.kml", "x")
OGR data source with driver: KML 
Source: "/tmp/x.kml", layer: "x"
with 1 features and 2 fields
Feature type: wkbPolygon with 2 dimensions

works nicely.

Now, you can try and get the name by parsing the KML as XML using an R XML parser, or you can maybe try reading it in R as a text file until you find the name tag.

The other approach is to run the command-line ogrinfo program which spits out the layer names of a KML file:

$ ogrinfo /tmp/x.kml 
Had to open data source read-only.
INFO: Open of `/tmp/x.kml'
      using driver `KML' successful.
1: x (Polygon)

here showing there is a polygon layer called x.

share|improve this answer
    
Thanks for your answer Spaced - solved the problem straight away. It's clear explanation like this that makes me love stack exchange! One 'bonus point' question: could I use the same command to read in a subset of the data (e.g. the first 1 million polygons)? Otherwise will look to split up the huge kmls with an external program. – RobinLovelace Apr 16 '13 at 14:10
2  
KML being XML isn't really designed for random access. The real solution is to put your spatial data into a spatial database, and have some spatial indexes for speed. Check out PostGIS. – Spacedman Apr 16 '13 at 14:33
    
OK good plan - I have told the client that PostGIS is the way forward for such big data, and am convinced that it's the right option for the kind of things he wants to do. Good excuse for me to learn it properly! – RobinLovelace Apr 16 '13 at 14:53
    
There is also the spatial extension to sqlite, a file based database, which would't require you to install a service and requires less configuration than PostGIS. – Frank Feb 10 at 8:32

If you want to do the alternative way using maptool, this should work:

tkml <- getKMLcoordinates(kmlfile="yourkml.kml", ignoreAltitude=T)
#make polygon
p1 = Polygon(tkml)
#make Polygon class
p2 = Polygons(list(p1), ID = "drivetime")
#make spatial polygons class
p3= SpatialPolygons(list(p2),proj4string=CRS("+init=epsg:4326"))

The key here is you need to go through a couple steps to make spatial polygon class.

share|improve this answer
    
hi @Seen, I've tried your approach but it seems not to work? I have a error: Error in Polygon(tkml) : coords must be a two-column matrix > head(tkml) [[1]] [1] -87.88141 30.49800 adn I have itas a list.. do you think its ok convert list of coordinates to matrix? tahnks ! – maycca Apr 22 at 21:16

Don't know if this is still a problem for anybody else, but I was running in circles for a while with this. What finally worked for me is below. It uses the XML package to get at the xmlValue of the right node. I had to set the layer parameter of readOGR to the name of the one of the folders within the kml file. When I set the layer parameter to the of the kml file, I would get the same error that RobinLovelace described above.

Shown below are a lot of code lines that only show how to see the various node levels of the kml document. I think this will be slightly different depending on the source of the kml. But you should be able to use the same logic to determine the correct parameter value.

Also, I created a list of kml files so it could be easily made into a function that could be put in an lapply - do.call pair. This could then pull in a data from a long list of kml files. Or, a lot of subfolders within a single kml file as it seems readOGR cannot deal with multiple subfolders in a kml file.

library(rgdal); library(XML)

# SET WORKING DIRECTORY FIRST!!
dir <- getwd()

kmlfilelist <- list.files(dir, pattern =".kml$", full.names=TRUE, recursive=FALSE)

doc0 <- xmlTreeParse(kmlfilelist[2], useInternal = TRUE)
rootNode0 <- xmlRoot(doc0)
rootName0 <- xmlName(rootNode0)
element1Name0 <- names(rootNode0)

nodeNames <- names(rootNode0[1][[1]])

# entire rootNode - kml Document level
rootNode0[[1]]

# 1st element of rootNode - kml file name
rootNode0[[1]][[1]] 

# 2nd element of rootNode - kml Style Map 
rootNode0[[1]][[2]] 

# 3rd element of rootNode - Style
rootNode0[[1]][[3]]

# 4th element of rootNode - Style
rootNode0[[1]][[4]] 

# 5th element of rootNode - kml Folder with data in it.
rootNode0[[1]][[5]] 

# 5th element 1st subelement of rootNode - kml Folder name with data in it. 
#  What to set readOGR() layer parameter to.
rootNode0[[1]][[5]][[1]] 

kmlfoldername <- xmlValue(rootNode0[[1]][[5]][[1]]) # Folder name to set = layer.

readOGR(dsn=kmlfilelist[2], layer =  kmlfoldername)
share|improve this answer

Don't know if I should have modified my previous answer. Perhaps, but that covers some things not in this answer, so I decided to leave it.

Anyway, the code below works well for me. It looks for all of xmlNodes in the kml file that are called "Folder" and then sets the layer parameter of readOGR to that xmlValue. Tested on working directory with around 6 separate kml files. Output is a list of imported SpatialDataFrames objects. Each SpatialDataFrame can easily be subset from the list.

Still does not address kml files with multiple Folder nodes. But that feature could easily be added with another nested apply function.

library(rgdal); library(XML)

# SET WORKING DIRECTORY FIRST!!
dir <- getwd()

kmlfilelist <- list.files(dir, pattern =".kml$", full.names=TRUE, recursive=FALSE)

ImportKml <- function (kmlfile) {
  doc0 <- xmlTreeParse(kmlfile, useInternal = TRUE)
  rootNode0 <- xmlRoot(doc0)
  rootName0 <- xmlName(rootNode0)
  element1Name0 <- names(rootNode0)

  kmlNodeNames <- unname(names(rootNode0[1][[1]]))
  kmlFolderNodeNum <- which(kmlNodeNames == "Folder")
  kmlFolderNodeName <- xmlValue(rootNode0[[1]][[kmlFolderNodeNum]][[1]])

  kmlIn <- readOGR(dsn=kmlfile, layer = kmlFolderNodeName)
}
ImportedKmls <- lapply(kmlfilelist, ImportKml)
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.