Processing a very large hosted KML file to identify what makes it so big [closed]

Question

I've run into a problem where I have two very large layers that I'm tasked to overlay on a map, showing cellular coverage across a the whole South Africa.

Layer 1 was given to me in various formats and it was really easy to work via the Javascript API after getting it into a fusion table.

Layer 2 is my struggle currently, as it was given to me as a hosted KML that is updated automatically every 24 hours. The only reason it's a problem is because of its size, being 90MB!

I've explored several options including:

• Downloading and simplifying with ogr2ogr, but it drops the size with about 20MB's, which is still nowhere near ideal.
• QGIS - trying to work with the layer hangs the system
• pyKML and fastKML - both seem to be capable of reading from it but I'm not sure if this is the best option. I like working in Python and not afraid of a learning curve in parsing the correct data, but I haven't been able to find a clear example of this use case.
• Google Earth - Seems to be too big even for GE as it also doesn't show up or hangs the program.

I'd like to know if there's a way to simplify such a big file. Right now I can open it in QGIS and can see Layer 2 there, but like I said, trying to work with it is memory intensive.

Can it be that the KML contains data that wasn't optimized for such a task? I'd like to be able to look at the KML data in a text editor, but even that hangs my Atom editor and others I've tried. I want to see what I'm working with in order to try and remove unnecessary stuff in there.

The url is here: http://siteadmin.wbs.co.za/kml/coverage.kml

I just need a few suggestions on how to handle this.

Answer 1

The file is big because it is a huge MultiPolygon (with 160,000 member Polygons) of pixel-ish structures, looking like the result of a raster to vector conversion. So, how to get the same data but in a smaller file:

• Ask the provider to provide you with a georeferenced raster format instead, such as GeoTIFF. There is just a single attribute Name: "Coverage: 20180927" so the data could be saved as 1 bit. Add reasonable compression and I would guess that it would not even be a Megabyte.

• Ask the provider to limit the coordinate precision to 6 or less. This should reduce the filesize by about one third with no loss of quality (if done correctly).

• Ask the provider to do the raster to vector conversion with a raster that does not have multiple values in it. That would collapse/union a lot of the parts into bigger ones. This should significantly reduce the complexity and probably at least halve the filesize.

• Ask the provider to enable gzip compression for KML files on their server. That would reduce the transfer to 25 Megabytes even without any other modifications.

For general usage of the file in editors adding linebreaks can make it more digestable (add one after each >, eg with sed 's#>#>\n#g'). For GIS it becomes easy to use one "exploded" into one feature per member polygon which speeds up display and analysis of subregions as not everything has to be loaded/considered all the time.

Answer 2

There's many ways to approach this. I personally would do it like this:

• Create a process that downloads data after it has been updated (e.g. CronJob on Linux server)
• Create a script (e.g. ogr2ogr) that saves it into a database, such as PostgreSQL (+PostGIS)
• Use a publishing service, such as Geoserver or Mapserver to create a layer or WMS from the database and also updates itself
• Embed this layer into a webmap with a framework of your choice (e.g. Leaflet, OpenLayers, Mapbox GL, Google Maps)