0

I've run into a problem where I have two very large layers that I'm tasked to overlay on a map, showing cellular coverage across a the whole South Africa.

Layer 1 was given to me in various formats and it was really easy to work via the Javascript API after getting it into a fusion table.

Layer 2 is my struggle currently, as it was given to me as a hosted KML that is updated automatically every 24 hours. The only reason it's a problem is because of its size, being 90MB!

I've explored several options including:

  • Downloading and simplifying with ogr2ogr, but it drops the size with about 20MB's, which is still nowhere near ideal.
  • QGIS - trying to work with the layer hangs the system
  • pyKML and fastKML - both seem to be capable of reading from it but I'm not sure if this is the best option. I like working in Python and not afraid of a learning curve in parsing the correct data, but I haven't been able to find a clear example of this use case.
  • Google Earth - Seems to be too big even for GE as it also doesn't show up or hangs the program.

I'd like to know if there's a way to simplify such a big file. Right now I can open it in QGIS and can see Layer 2 there, but like I said, trying to work with it is memory intensive.

Can it be that the KML contains data that wasn't optimized for such a task? I'd like to be able to look at the KML data in a text editor, but even that hangs my Atom editor and others I've tried. I want to see what I'm working with in order to try and remove unnecessary stuff in there.

The url is here: http://siteadmin.wbs.co.za/kml/coverage.kml

I just need a few suggestions on how to handle this.

closed as too broad by Jochen Schwarze, Vince, Mapperz Sep 28 '18 at 14:50

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Welcome to GIS SE! As a new user be sure to take the Tour to learn about our focussed Q&A format. I would try loading your KML into a file geodatabase using ArcGIS Pro's KML To Layer tool. – PolyGeo Sep 28 '18 at 6:39
  • Could you specify what exactly the problem/s is/are? What do you need to achieve exactly in the end? – bugmenot123 Sep 28 '18 at 7:32
  • The main problem is learning the best solution for parsing large data sets like this. I'm sure I can figure it out by looking at the data in a text editor, but the file is so big that it crashes any editor I use to open it. Does that make sense? – boognish Sep 28 '18 at 7:39
  • What is the parsing exactly that you want? Best solution for what? OGR can easily digest the file. You can easily process it with GNU coreutils. – bugmenot123 Sep 28 '18 at 9:21
  • Sorry if I'm not clear. The best solution for establishing what exactly is making this file to be so big and if I'm able to remove unnecessary data or not. I need to establish if a 90MB KML file is okay to be left as a hosted source or if I should urge the client to provide another more compact dataset. I will have a good look at OGR as I'm comfortable on the Linux command line. – boognish Sep 28 '18 at 9:33
0

screenshot

The file is big because it is a huge MultiPolygon (with 160,000 member Polygons) of pixel-ish structures, looking like the result of a raster to vector conversion. So, how to get the same data but in a smaller file:

  • Ask the provider to provide you with a georeferenced raster format instead, such as GeoTIFF. There is just a single attribute Name: "Coverage: 20180927" so the data could be saved as 1 bit. Add reasonable compression and I would guess that it would not even be a Megabyte.

  • Ask the provider to limit the coordinate precision to 6 or less. This should reduce the filesize by about one third with no loss of quality (if done correctly).

  • Ask the provider to do the raster to vector conversion with a raster that does not have multiple values in it. That would collapse/union a lot of the parts into bigger ones. This should significantly reduce the complexity and probably at least halve the filesize.

  • Ask the provider to enable gzip compression for KML files on their server. That would reduce the transfer to 25 Megabytes even without any other modifications.

For general usage of the file in editors adding linebreaks can make it more digestable (add one after each >, eg with sed 's#>#>\n#g'). For GIS it becomes easy to use one "exploded" into one feature per member polygon which speeds up display and analysis of subregions as not everything has to be loaded/considered all the time.

  • 1
    Thank you! This makes sense and was the kind of information I was after. I'm going to work through these points with the provider. – boognish Sep 28 '18 at 10:06
  • Glad to help! :) – bugmenot123 Sep 28 '18 at 11:16
0

There's many ways to approach this. I personally would do it like this:

  • Create a process that downloads data after it has been updated (e.g. CronJob on Linux server)
  • Create a script (e.g. ogr2ogr) that saves it into a database, such as PostgreSQL (+PostGIS)
  • Use a publishing service, such as Geoserver or Mapserver to create a layer or WMS from the database and also updates itself
  • Embed this layer into a webmap with a framework of your choice (e.g. Leaflet, OpenLayers, Mapbox GL, Google Maps)
  • This doesn't appear to answer the question. – Vince Sep 28 '18 at 11:40
  • my intention is to solve his problem: overlay huge data on his own map and not how to minify KML data – Revo Sep 28 '18 at 12:16
  • Thank you @Revo. Though it doesn't answer my question explicitly I appreciate the process outlined and will definitely explore the options you mentioned. – boognish Sep 28 '18 at 14:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.