Does anyone have information about possible formats for highly compressed vector data?

Some basic requirements.

  • Must be possible to read using the larger commercially available platforms. (ArcGIS, Luciad, MapInfo etc.)
  • Must have tools that allows us to create files/format from other data formats (i.e. Shape)
  • Speed when reading the format is important as the data is among other things used as a live map in mobile platforms. Compression speed is less important (although nice)
  • Must be possible to use a map product in a wide scale range, a 50k map shall be possible to read at a 1 000 k zoom level
  • Ability to do selective updates on area, feature-ID or other search criteria would be nice, but not a must.
  • Memory footprint must be small, it needs to work with Windows XP and/or Windows 7 allocating less than 1GB RAM so some kind of smart cache management must be implemented

To give an idea of the required compression and as input for memory footprint requirements, I need to store approximately 1TB of shape data to a 100GB disk.

Does anyone have suggestions for good products?

  • Do you need "online access" for the full data set? shapefiles will (usually) zip very well, but that obviously isn't seekable.
    – BradHards
    Aug 10 '12 at 9:41
  • Yes the whole data set need to be "online" and accessible to the applications all the time. Zip or other offline compression systems are not feasible. Aug 10 '12 at 9:45
  • Re-reading your post, I'm slightly confused as to whether you want a tool or a file format. Perhaps it might be best to describe the system you're designing, rather than the solutions you think you might need. Aug 10 '12 at 10:40
  • OK, sorry that I'm unclear. I'll try to explain what I wanta little better. I need a way to store and access large amount of vector data. The point about the tools is to make sure I don't get a transport format (zip) that needs to be uncompressed before it can be used. The data will be used in a variety of situations and we sould like to minimize the number of formats we distribute. One application is vehicle mounted systems (stand alone) with limited disk space and memory. Aug 10 '12 at 10:51

SpatiaLite does support compressed geometry, but you will not get 10:1 compression out of it.

Your real compression performance will probably depend a lot on the data you are putting in. Is it all geometries? If you have a lot of non-spatial data (or a lot of text attributes for spatial points), then it doesn't really matter what you do the geometries - you need to find some way to compress that data instead.

  • Thanks! I had a look at the site and it looks lite it doesn't interface with any major GIS-software. Maybe I'm missing something on their pages? We have a range of existing applications we need to supply data for and they mostly use ArcGIS or some other commercial toolkit to read and display the data. I need to find a format that can be read natively or that there is a plugin for. Aug 10 '12 at 14:05
  • Support in ArcGIS is limited - perhaps FME, although I don't have either of these. QGIS supports it very well.
    – BradHards
    Aug 10 '12 at 23:13

As others have said, I think you are going to struggle to find a format that meets your compression requirements. You would have to create your own custom format, which given your requirement to use commercial software is not going to be viable.

I think you need to possibly first consider how you can make your data models more efficient, then look at the compression aspects. For example, do you have a lot of repetition of geometry? You could then have a base set of geometry layers with unique IDs and then separate attribute data sets that reference the geometry by ID - that way you can have multiple views of the same geometry serving specific functions. Most decent software packages will then allow you to create joins or relates in order to create the unified view for a layer.

GML is a good example of a format that supports this kind of relational data model, though being a verbose format file sizes will be large. You can compress GML using gzip compression and can potentially get a 20:1 ratio but then you are relying on the software being able to support compressed GML.

Regardless, I would urge you to first look at your data models and see where there could be savings to be had. FME from Safe Software is your best bet if you need to start manipulating your data models.


Looking here, there appear to be no compressed vector formats that might be commonly supported (except formats like KMZ, but that's unlikely to compress to a size any different to a binary format).

You are very unlikely to achieve a 10:1 compression ratio of shapefile data, which although isn't the most space-efficient format, will only give you about a 3:1 ratio using gzip or similar.

To achieve that sort of ratio, you could use some sort of lossy compression, but I don't know of anything that uses it, and although I have a couple of ideas on how one might implement it, it would be far from standard. It would be much much cheaper to kit your server out with a 1TB disk than to spend time and money developing a custom solution.

You are also confusing data storage with data representation. Your 4th point mentions being able to view the data at different scales, but this is a function of your renderer, not the format per se. Again, a hypothetical lossily compressed file could store data at various resolutions in a sort of LoD structure, but that is likely to increase data size if anything.

If your data is to be on a server somewhere accessible by mobile applications, you're far better off using existing tools that have been designed for the purpose. A WFS server (such as GeoServer or MapServer) is ideally suited to this sort of application. The client makes a request for data of a specific area, normally that covered by the screen, and the WFS sends vector data for just that area, so all the heavy lifting is done by the server. It's then up to the application to render that data. An alternative would be to use the WMS features of MapServer and GeoServer, in which all the rendering is done by the server, and then it sends an image tile to the client. This enables features such as server-side caching of tiles, as well as scale-dependent rendering, with the minimum of work by you. They both read myriad formats, so you can author your data exactly how you like, and store it where you like, and they do all the cool stuff. Quantum GIS also has a WMS server, so you can author and serve data all in the same application.

  • Ok, thanks for the answer! As you point out the scale range requirement probably requires som kind of LoD-info in the data set. More disk or a WMS solution will not work because som of the systems will be stand alone with limited hardware resources. Aug 10 '12 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.